[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:53:09 dist-test-slave-49h3 krb5kdc[31599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:53:09 dist-test-slave-49h3 krb5kdc[31599](info): set up 2 sockets
Aug 06 10:53:09 dist-test-slave-49h3 krb5kdc[31599](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 10:53:10.670038 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:53:10 dist-test-slave-49h3 krb5kdc[31599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:53:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:53:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:53:10.887786 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:35157
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:35157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:11.237365 31613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:11.237772 31613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:11.237907 31613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:11.245600 31613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:53:11.245740 31613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:11.245815 31613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:11.245899 31613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:53:11.245960 31613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:53:11.256469 31613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:35157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:35157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:11.257905 31613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:11.259209 31613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:11.270826 31618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:11.270854 31619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:11.270785 31621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:11.271955 31613 server_base.cc:1034] running on GCE node
I20260806 10:53:11.272646 31613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:11.274010 31613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:11.275231 31613 hybrid_clock.cc:648] HybridClock initialized: now 1786013591275226 us; error 81 us; skew 500 ppm
I20260806 10:53:11.275595 31613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:11 dist-test-slave-49h3 krb5kdc[31599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013591, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:11.279063 31613 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:53:11.280696 31613 webserver.cc:460] Webserver started at http://127.30.218.62:37975/ using document root <none> and password file <none>
I20260806 10:53:11.281502 31613 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:11.281635 31613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:11.281956 31613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:11.285383 31613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "579e9ff2792c4e0d8f3bfb2f5ca365c7"
format_stamp: "Formatted at 2026-08-06 10:53:11 on dist-test-slave-49h3"
server_key: "4f6f92b923df9608cc385e213a58d32b"
server_key_iv: "444b918b7211a683433b353979ab8baf"
server_key_version: "encryptionkey@0"
I20260806 10:53:11.286324 31613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "579e9ff2792c4e0d8f3bfb2f5ca365c7"
format_stamp: "Formatted at 2026-08-06 10:53:11 on dist-test-slave-49h3"
server_key: "4f6f92b923df9608cc385e213a58d32b"
server_key_iv: "444b918b7211a683433b353979ab8baf"
server_key_version: "encryptionkey@0"
I20260806 10:53:11.292212 31613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260806 10:53:11.296200 31628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:11.297735 31613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 10:53:11.297932 31613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "579e9ff2792c4e0d8f3bfb2f5ca365c7"
format_stamp: "Formatted at 2026-08-06 10:53:11 on dist-test-slave-49h3"
server_key: "4f6f92b923df9608cc385e213a58d32b"
server_key_iv: "444b918b7211a683433b353979ab8baf"
server_key_version: "encryptionkey@0"
I20260806 10:53:11.298143 31613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:11.324649 31613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:11.329893 31613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:11.330224 31613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:11.349747 31613 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:35157
I20260806 10:53:11.349773 31681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:35157 every 8 connection(s)
I20260806 10:53:11.351136 31613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:53:11.355007 31682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:11.361419 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 31613
I20260806 10:53:11.361865 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:53:11.364607 31592 external_mini_cluster.cc:1428] Setting key 6545b89309f5bc22e612740b1072f901
I20260806 10:53:11.368407 31682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 579e9ff2792c4e0d8f3bfb2f5ca365c7: Bootstrap starting.
I20260806 10:53:11.372618 31682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 579e9ff2792c4e0d8f3bfb2f5ca365c7: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:11.374209 31682 log.cc:824] T 00000000000000000000000000000000 P 579e9ff2792c4e0d8f3bfb2f5ca365c7: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:11.378043 31682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 579e9ff2792c4e0d8f3bfb2f5ca365c7: No bootstrap required, opened a new log
I20260806 10:53:11.390209 31682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 579e9ff2792c4e0d8f3bfb2f5ca365c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579e9ff2792c4e0d8f3bfb2f5ca365c7" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 35157 } }
I20260806 10:53:11.390632 31682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 579e9ff2792c4e0d8f3bfb2f5ca365c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:11.390758 31682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 579e9ff2792c4e0d8f3bfb2f5ca365c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 579e9ff2792c4e0d8f3bfb2f5ca365c7, State: Initialized, Role: FOLLOWER
I20260806 10:53:11.391405 31682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 579e9ff2792c4e0d8f3bfb2f5ca365c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579e9ff2792c4e0d8f3bfb2f5ca365c7" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 35157 } }
I20260806 10:53:11.391728 31682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 579e9ff2792c4e0d8f3bfb2f5ca365c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:53:11.391881 31682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 579e9ff2792c4e0d8f3bfb2f5ca365c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:53:11.392081 31682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 579e9ff2792c4e0d8f3bfb2f5ca365c7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:11.395149 31682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 579e9ff2792c4e0d8f3bfb2f5ca365c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579e9ff2792c4e0d8f3bfb2f5ca365c7" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 35157 } }
I20260806 10:53:11.395907 31682 leader_election.cc:302] T 00000000000000000000000000000000 P 579e9ff2792c4e0d8f3bfb2f5ca365c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 579e9ff2792c4e0d8f3bfb2f5ca365c7; no voters:
I20260806 10:53:11.396688 31682 leader_election.cc:288] T 00000000000000000000000000000000 P 579e9ff2792c4e0d8f3bfb2f5ca365c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:53:11.396793 31687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 579e9ff2792c4e0d8f3bfb2f5ca365c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:11.398046 31687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 579e9ff2792c4e0d8f3bfb2f5ca365c7 [term 1 LEADER]: Becoming Leader. State: Replica: 579e9ff2792c4e0d8f3bfb2f5ca365c7, State: Running, Role: LEADER
I20260806 10:53:11.398732 31687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 579e9ff2792c4e0d8f3bfb2f5ca365c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579e9ff2792c4e0d8f3bfb2f5ca365c7" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 35157 } }
I20260806 10:53:11.399044 31682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 579e9ff2792c4e0d8f3bfb2f5ca365c7 [sys.catalog]: configured and running, proceeding with master startup.
Aug 06 10:53:11 dist-test-slave-49h3 krb5kdc[31599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:11.404238 31688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 579e9ff2792c4e0d8f3bfb2f5ca365c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "579e9ff2792c4e0d8f3bfb2f5ca365c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579e9ff2792c4e0d8f3bfb2f5ca365c7" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 35157 } } }
I20260806 10:53:11.404668 31688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 579e9ff2792c4e0d8f3bfb2f5ca365c7 [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:11.404073 31689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 579e9ff2792c4e0d8f3bfb2f5ca365c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 579e9ff2792c4e0d8f3bfb2f5ca365c7. Latest consensus state: current_term: 1 leader_uuid: "579e9ff2792c4e0d8f3bfb2f5ca365c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579e9ff2792c4e0d8f3bfb2f5ca365c7" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 35157 } } }
I20260806 10:53:11.405296 31689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 579e9ff2792c4e0d8f3bfb2f5ca365c7 [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:11.417178 31698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:53:11.417619 31684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:11.369854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47458 (local address 127.30.218.62:35157)
0806 10:53:11.372720 (+ 2866us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:11.372734 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:11.372798 (+ 64us) server_negotiation.cc:407] Connection header received
0806 10:53:11.384497 (+ 11699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:11.384522 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:11.384996 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:11.385362 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:11.389782 (+ 4420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:11.391334 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:11.394013 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:11.394744 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:11.394888 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:11.403819 (+ 8931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:11.403836 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:11.403863 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:11.403939 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:11.409594 (+ 5655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:11.411082 (+ 1488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:11.411094 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:11.411103 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:11.411401 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:11.412105 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:11.412111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:11.412115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:11.413002 (+ 887us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:11.413475 (+ 473us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:11.414988 (+ 1513us) server_negotiation.cc:299] Negotiation successful
0806 10:53:11.415555 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2620,"thread_start_us":2341,"threads_started":1}
I20260806 10:53:11.422962 31698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:53:11.432057 31698 catalog_manager.cc:1347] Generated new cluster ID: 431f8ba763ca4fd4901ba225f79170ee
I20260806 10:53:11.432202 31698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:53:11.443723 31698 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:53:11.445819 31698 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:53:11.456996 31698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 579e9ff2792c4e0d8f3bfb2f5ca365c7: Generated new TSK 0
I20260806 10:53:11.458014 31698 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
I20260806 10:53:11.515403 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:35157
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:11.863408 31708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:11.863718 31708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:11.863853 31708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:11.871589 31708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:11.871728 31708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:11.871901 31708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:53:11.882428 31708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:35157
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:11.883816 31708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:11.885284 31708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:11.900072 31714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:11.900041 31713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:11.900226 31716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:11.900931 31708 server_base.cc:1034] running on GCE node
I20260806 10:53:11.901666 31708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:11.902743 31708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:11.903947 31708 hybrid_clock.cc:648] HybridClock initialized: now 1786013591903888 us; error 52 us; skew 500 ppm
I20260806 10:53:11.904284 31708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:11 dist-test-slave-49h3 krb5kdc[31599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013591, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:11.907660 31708 init.cc:377] Logged in from keytab as kudu/127.30.218.1@KRBTEST.COM (short username kudu)
I20260806 10:53:11.909330 31708 webserver.cc:460] Webserver started at http://127.30.218.1:45959/ using document root <none> and password file <none>
I20260806 10:53:11.910137 31708 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:11.910274 31708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:11.910614 31708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:11.913940 31708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "00f618c7526c47c480066ed91562417a"
format_stamp: "Formatted at 2026-08-06 10:53:11 on dist-test-slave-49h3"
server_key: "8a78a38a0d0a9e4c52f16dab52ee52e6"
server_key_iv: "ec88fdeb844543ed56f5714fc2bfef91"
server_key_version: "encryptionkey@0"
I20260806 10:53:11.914791 31708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "00f618c7526c47c480066ed91562417a"
format_stamp: "Formatted at 2026-08-06 10:53:11 on dist-test-slave-49h3"
server_key: "8a78a38a0d0a9e4c52f16dab52ee52e6"
server_key_iv: "ec88fdeb844543ed56f5714fc2bfef91"
server_key_version: "encryptionkey@0"
I20260806 10:53:11.920747 31708 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 10:53:11.924803 31723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:11.926398 31708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:53:11.926587 31708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "00f618c7526c47c480066ed91562417a"
format_stamp: "Formatted at 2026-08-06 10:53:11 on dist-test-slave-49h3"
server_key: "8a78a38a0d0a9e4c52f16dab52ee52e6"
server_key_iv: "ec88fdeb844543ed56f5714fc2bfef91"
server_key_version: "encryptionkey@0"
I20260806 10:53:11.926838 31708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:11.975728 31708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:11.980041 31708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:11.980346 31708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:11.981380 31708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:53:11.982892 31708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:11.982975 31708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:11.983103 31708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:11.983157 31708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:12.015928 31708 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:46299
I20260806 10:53:12.015970 31839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:46299 every 8 connection(s)
I20260806 10:53:12.017318 31708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:53:12.018941 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 31708
I20260806 10:53:12.019181 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:53:12.019858 31592 external_mini_cluster.cc:1428] Setting key a05289a02720b46678db478178c478cc
Aug 06 10:53:12 dist-test-slave-49h3 krb5kdc[31599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013591, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:12.040087 31842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:12.020234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:40155 (local address 127.30.218.62:35157)
0806 10:53:12.020887 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:12.020893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:12.021938 (+ 1045us) server_negotiation.cc:407] Connection header received
0806 10:53:12.023978 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:12.023985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:12.024096 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:12.024238 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:12.026012 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:12.027015 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:12.028679 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:12.029245 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:12.029385 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:12.033542 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:12.033548 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:12.033551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:12.033592 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:12.036282 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:12.037291 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:12.037298 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:12.037302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:12.037527 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:12.038009 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:12.038014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:12.038018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:12.038436 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:12.038611 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:12.039379 (+ 768us) server_negotiation.cc:299] Negotiation successful
0806 10:53:12.039659 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":375,"threads_started":1}
I20260806 10:53:12.040984 31843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:12.020871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:35157 (local address 127.30.218.1:40155)
0806 10:53:12.021749 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:12.021813 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:12.023650 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:12.024494 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:12.024504 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:12.025078 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:12.025809 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:12.025830 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:12.027154 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:12.027157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:12.028509 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:12.028524 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:12.029510 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:12.029519 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:12.029775 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:12.030545 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:12.030570 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:12.033229 (+ 2659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:12.036488 (+ 3259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:12.036498 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:12.036515 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:12.037076 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:12.037642 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:12.037646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:12.037649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:12.037872 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:12.038555 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:12.038562 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:12.038962 (+ 400us) client_negotiation.cc:769] Sending connection context
0806 10:53:12.039216 (+ 254us) client_negotiation.cc:240] Negotiation successful
0806 10:53:12.039677 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":311,"threads_started":1}
WARNING: no policy specified for kudu/127.30.218.2@KRBTEST.COM; defaulting to no policy
I20260806 10:53:12.042750 31840 heartbeater.cc:347] Connected to a master server at 127.30.218.62:35157
I20260806 10:53:12.043071 31840 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:12.043999 31840 heartbeater.cc:511] Master 127.30.218.62:35157 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.2@KRBTEST.COM" created.
I20260806 10:53:12.046397 31646 ts_manager.cc:194] Registered new tserver with Master: 00f618c7526c47c480066ed91562417a (127.30.218.1:46299)
I20260806 10:53:12.047955 31646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.1@KRBTEST.COM'} at 127.30.218.1:40155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
I20260806 10:53:12.068372 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:35157
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:12.410161 31846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:12.410476 31846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:12.410575 31846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:12.418632 31846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:12.418759 31846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:12.418888 31846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:53:12.429280 31846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:35157
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:12.430709 31846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:12.432154 31846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:12.446439 31854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:12.446483 31851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:12.446398 31852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:12.447448 31846 server_base.cc:1034] running on GCE node
I20260806 10:53:12.448117 31846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:12.449119 31846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:12.450330 31846 hybrid_clock.cc:648] HybridClock initialized: now 1786013592450278 us; error 41 us; skew 500 ppm
I20260806 10:53:12.450675 31846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:12 dist-test-slave-49h3 krb5kdc[31599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013592, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:12.454142 31846 init.cc:377] Logged in from keytab as kudu/127.30.218.2@KRBTEST.COM (short username kudu)
I20260806 10:53:12.455730 31846 webserver.cc:460] Webserver started at http://127.30.218.2:34061/ using document root <none> and password file <none>
I20260806 10:53:12.456492 31846 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:12.456621 31846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:12.457039 31846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:12.460309 31846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "36c3ac1d4bef442f94adbf584a1d5824"
format_stamp: "Formatted at 2026-08-06 10:53:12 on dist-test-slave-49h3"
server_key: "77f4353843944d893fbd1d2f4f91c731"
server_key_iv: "0541c7e0fc1f67a8b10ebf5abfbb51c2"
server_key_version: "encryptionkey@0"
I20260806 10:53:12.461155 31846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "36c3ac1d4bef442f94adbf584a1d5824"
format_stamp: "Formatted at 2026-08-06 10:53:12 on dist-test-slave-49h3"
server_key: "77f4353843944d893fbd1d2f4f91c731"
server_key_iv: "0541c7e0fc1f67a8b10ebf5abfbb51c2"
server_key_version: "encryptionkey@0"
I20260806 10:53:12.467163 31846 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260806 10:53:12.471216 31861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:12.472735 31846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 10:53:12.472920 31846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "36c3ac1d4bef442f94adbf584a1d5824"
format_stamp: "Formatted at 2026-08-06 10:53:12 on dist-test-slave-49h3"
server_key: "77f4353843944d893fbd1d2f4f91c731"
server_key_iv: "0541c7e0fc1f67a8b10ebf5abfbb51c2"
server_key_version: "encryptionkey@0"
I20260806 10:53:12.473146 31846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:12.502492 31846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:12.507265 31846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:12.507556 31846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:12.508603 31846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:53:12.510140 31846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:12.510221 31846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:12.510314 31846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:12.510358 31846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:12.549019 31846 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:46757
I20260806 10:53:12.549085 31976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:46757 every 8 connection(s)
I20260806 10:53:12.550894 31846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:53:12.560279 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 31846
I20260806 10:53:12.560519 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:53:12.561152 31592 external_mini_cluster.cc:1428] Setting key 5dde1f1269be67a31597370565bbed1b
Aug 06 10:53:12 dist-test-slave-49h3 krb5kdc[31599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013592, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:12.577323 31979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:12.554083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:35629 (local address 127.30.218.62:35157)
0806 10:53:12.554629 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:12.554635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:12.556282 (+ 1647us) server_negotiation.cc:407] Connection header received
0806 10:53:12.558937 (+ 2655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:12.558942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:12.559037 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:12.559147 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:12.561403 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:12.562432 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:12.564870 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:12.565319 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:12.565441 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:12.569872 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:12.569880 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:12.569885 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:12.569934 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:12.573290 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:12.574418 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:12.574425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:12.574430 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:12.574603 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:12.575164 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:12.575170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:12.575173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:12.575496 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:12.575651 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:12.576622 (+ 971us) server_negotiation.cc:299] Negotiation successful
0806 10:53:12.576886 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":263,"threads_started":1}
I20260806 10:53:12.578930 31980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:12.554760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:35157 (local address 127.30.218.2:35629)
0806 10:53:12.555755 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:12.555827 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:12.558409 (+ 2582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:12.559479 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:12.559490 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:12.560295 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:12.561154 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:12.561177 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:12.562610 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:12.562632 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:12.564695 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:12.564721 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:12.565604 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:12.565611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:12.565945 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:12.566831 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:12.566865 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:12.569546 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:12.573523 (+ 3977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:12.573543 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:12.573566 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:12.574277 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:12.574736 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:12.574740 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:12.574744 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:12.575033 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:12.575643 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:12.575652 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:12.576162 (+ 510us) client_negotiation.cc:769] Sending connection context
0806 10:53:12.576490 (+ 328us) client_negotiation.cc:240] Negotiation successful
0806 10:53:12.577223 (+ 733us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":392,"threads_started":1}
I20260806 10:53:12.580567 31977 heartbeater.cc:347] Connected to a master server at 127.30.218.62:35157
I20260806 10:53:12.580947 31977 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:12.581820 31977 heartbeater.cc:511] Master 127.30.218.62:35157 requested a full tablet report, sending...
I20260806 10:53:12.583403 31646 ts_manager.cc:194] Registered new tserver with Master: 36c3ac1d4bef442f94adbf584a1d5824 (127.30.218.2:46757)
I20260806 10:53:12.584261 31646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.2@KRBTEST.COM'} at 127.30.218.2:35629
WARNING: no policy specified for kudu/127.30.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
I20260806 10:53:12.614394 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:35157
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:12.961587 31983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:12.961918 31983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:12.962036 31983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:12.969834 31983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:12.969973 31983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:12.970098 31983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:53:12.980705 31983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:35157
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:12.982167 31983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:12.983539 31983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:12.997269 31989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:12.997309 31991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:12.997236 31988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:12.998487 31983 server_base.cc:1034] running on GCE node
I20260806 10:53:12.999199 31983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:13.000363 31983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:13.001562 31983 hybrid_clock.cc:648] HybridClock initialized: now 1786013593001507 us; error 45 us; skew 500 ppm
I20260806 10:53:13.001966 31983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:12 dist-test-slave-49h3 krb5kdc[31599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013592, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:13.005394 31983 init.cc:377] Logged in from keytab as kudu/127.30.218.3@KRBTEST.COM (short username kudu)
I20260806 10:53:13.007009 31983 webserver.cc:460] Webserver started at http://127.30.218.3:36907/ using document root <none> and password file <none>
I20260806 10:53:13.007735 31983 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:13.007856 31983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:13.008184 31983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:13.011359 31983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "6b19dc8ee98d4557baf99e0d86ac584a"
format_stamp: "Formatted at 2026-08-06 10:53:13 on dist-test-slave-49h3"
server_key: "781af415ab0100dab613edee888e306d"
server_key_iv: "a14b7a046d47f67aabfbdea2ab4efdf2"
server_key_version: "encryptionkey@0"
I20260806 10:53:13.012180 31983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "6b19dc8ee98d4557baf99e0d86ac584a"
format_stamp: "Formatted at 2026-08-06 10:53:13 on dist-test-slave-49h3"
server_key: "781af415ab0100dab613edee888e306d"
server_key_iv: "a14b7a046d47f67aabfbdea2ab4efdf2"
server_key_version: "encryptionkey@0"
I20260806 10:53:13.018698 31983 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260806 10:53:13.023500 31998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:13.025162 31983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 10:53:13.025344 31983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "6b19dc8ee98d4557baf99e0d86ac584a"
format_stamp: "Formatted at 2026-08-06 10:53:13 on dist-test-slave-49h3"
server_key: "781af415ab0100dab613edee888e306d"
server_key_iv: "a14b7a046d47f67aabfbdea2ab4efdf2"
server_key_version: "encryptionkey@0"
I20260806 10:53:13.025609 31983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:13.049726 31983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:13.051285 31840 heartbeater.cc:503] Master 127.30.218.62:35157 was elected leader, sending a full tablet report...
I20260806 10:53:13.054682 31983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:13.054952 31983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:13.055939 31983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:53:13.057281 31983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:13.057358 31983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:13.057469 31983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:13.057515 31983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 10:53:13.088954 31983 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:38539
I20260806 10:53:13.089018 32114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:38539 every 8 connection(s)
I20260806 10:53:13.090354 31983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:53:13.097368 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 31983
I20260806 10:53:13.097718 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:53:13.098340 31592 external_mini_cluster.cc:1428] Setting key 5230de3f812b2af09c39c7c4a2a41a47
Aug 06 10:53:13 dist-test-slave-49h3 krb5kdc[31599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013592, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:13.112720 32117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:13.092960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:41851 (local address 127.30.218.62:35157)
0806 10:53:13.093395 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:13.093399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:13.094357 (+ 958us) server_negotiation.cc:407] Connection header received
0806 10:53:13.096387 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:13.096391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:13.096486 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:13.096584 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:13.099050 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:13.100120 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:13.102203 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:13.102665 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:13.102786 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:13.106748 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:13.106754 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:13.106757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:13.106799 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:13.109578 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:13.110406 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:13.110410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:13.110412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:13.110546 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:13.111036 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:13.111040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:13.111042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:13.111339 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:13.111470 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:13.112232 (+ 762us) server_negotiation.cc:299] Negotiation successful
0806 10:53:13.112421 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":216,"threads_started":1}
I20260806 10:53:13.114037 32118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:13.093419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:35157 (local address 127.30.218.3:41851)
0806 10:53:13.094197 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:13.094237 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:13.096008 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:13.096821 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:13.096831 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:13.097839 (+ 1008us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:13.098736 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:13.098773 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:13.100315 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:13.100321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:13.102023 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:13.102043 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:13.102917 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:13.102926 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:13.103270 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:13.103955 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:13.103981 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:13.106462 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:13.109747 (+ 3285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:13.109756 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:13.109772 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:13.110294 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:13.110682 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:13.110687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:13.110689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:13.110932 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:13.111477 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:13.111485 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:13.111890 (+ 405us) client_negotiation.cc:769] Sending connection context
0806 10:53:13.112144 (+ 254us) client_negotiation.cc:240] Negotiation successful
0806 10:53:13.112611 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":278,"threads_started":1}
I20260806 10:53:13.115531 32115 heartbeater.cc:347] Connected to a master server at 127.30.218.62:35157
I20260806 10:53:13.115865 32115 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:13.116633 32115 heartbeater.cc:511] Master 127.30.218.62:35157 requested a full tablet report, sending...
I20260806 10:53:13.118291 31645 ts_manager.cc:194] Registered new tserver with Master: 6b19dc8ee98d4557baf99e0d86ac584a (127.30.218.3:38539)
I20260806 10:53:13.119247 31645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.3@KRBTEST.COM'} at 127.30.218.3:41851
I20260806 10:53:13.128479 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 10:53:13 dist-test-slave-49h3 krb5kdc[31599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013593, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 10:53:13 dist-test-slave-49h3 krb5kdc[31599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013593, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.218.1@KRBTEST.COM
I20260806 10:53:13.183727 32123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:13.167118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43084 (local address 127.30.218.1:46299)
0806 10:53:13.167821 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:13.167829 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:13.167859 (+ 30us) server_negotiation.cc:407] Connection header received
0806 10:53:13.168020 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:13.168034 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:13.168270 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:13.168411 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:13.170060 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:13.171139 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:13.172509 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:13.173166 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:13.173286 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:13.177705 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:13.177754 (+ 49us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:13.177778 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:13.177846 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:13.180836 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:13.181664 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:13.181671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:13.181674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:13.181832 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:13.182194 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:13.182198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:13.182199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:13.182609 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:13.182881 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:13.183098 (+ 217us) server_negotiation.cc:299] Negotiation successful
0806 10:53:13.183351 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":380,"threads_started":1}
W20260806 10:53:13.185127 31751 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43084
Aug 06 10:53:13 dist-test-slave-49h3 krb5kdc[31599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 10:53:13 dist-test-slave-49h3 krb5kdc[31599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.1@KRBTEST.COM
I20260806 10:53:13.237891 32123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:13.223455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43088 (local address 127.30.218.1:46299)
0806 10:53:13.223694 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:13.223701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:13.223972 (+ 271us) server_negotiation.cc:407] Connection header received
0806 10:53:13.224132 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:13.224136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:13.224234 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:13.224326 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:13.225790 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:13.226523 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:13.227738 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:13.228159 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:13.228275 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:13.232501 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:13.232507 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:13.232510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:13.232553 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:13.235333 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:13.236014 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:13.236018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:13.236020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:13.236158 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:13.236517 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:13.236520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:13.236522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:13.236795 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:13.236992 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:13.237306 (+ 314us) server_negotiation.cc:299] Negotiation successful
0806 10:53:13.237558 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 10:53:13.239531 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 31708
I20260806 10:53:13.253201 32123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:13.240586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43100 (local address 127.30.218.1:46299)
0806 10:53:13.240766 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:13.240772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:13.241073 (+ 301us) server_negotiation.cc:407] Connection header received
0806 10:53:13.241244 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:13.241250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:13.241316 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:13.241398 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:13.242705 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:13.243436 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:13.244768 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:13.245220 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:13.245333 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:13.247915 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:13.247920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:13.247924 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:13.247966 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:13.250399 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:13.251160 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:13.251166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:13.251169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:13.251335 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:13.251725 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:13.251731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:13.251733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:13.252027 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:13.252289 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:13.252586 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 10:53:13.252817 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 10:53:13.254055 31835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:13.339839 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 31708
I20260806 10:53:13.355585 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 31846
Aug 06 10:53:13 dist-test-slave-49h3 krb5kdc[31599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.2@KRBTEST.COM
I20260806 10:53:13.371205 32130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:13.356560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37284 (local address 127.30.218.2:46757)
0806 10:53:13.357086 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:13.357092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:13.357120 (+ 28us) server_negotiation.cc:407] Connection header received
0806 10:53:13.357178 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:13.357186 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:13.357386 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:13.357570 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:13.358659 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:13.359621 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:13.360777 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:13.361314 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:13.361407 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:13.365532 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:13.365549 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:13.365565 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:13.365620 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:13.368454 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:13.369142 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:13.369151 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:13.369154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:13.369293 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:13.369677 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:13.369680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:13.369682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:13.370066 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:13.370316 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:13.370607 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 10:53:13.370871 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":279,"threads_started":1}
I20260806 10:53:13.372563 31972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:13.456765 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 31846
I20260806 10:53:13.473047 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 31983
Aug 06 10:53:13 dist-test-slave-49h3 krb5kdc[31599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.3@KRBTEST.COM
I20260806 10:53:13.490088 32132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:13.473982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41676 (local address 127.30.218.3:38539)
0806 10:53:13.474568 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:13.474575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:13.474605 (+ 30us) server_negotiation.cc:407] Connection header received
0806 10:53:13.474670 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:13.474676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:13.474888 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:13.475047 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:13.476340 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:13.477391 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:13.478754 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:13.479367 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:13.479485 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:13.483827 (+ 4342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:13.483842 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:13.483859 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:13.483900 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:13.486893 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:13.487706 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:13.487715 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:13.487717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:13.487873 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:13.488287 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:13.488291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:13.488294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:13.488691 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:13.489025 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:13.489388 (+ 363us) server_negotiation.cc:299] Negotiation successful
0806 10:53:13.489725 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":283,"threads_started":1}
I20260806 10:53:13.491402 32109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:13.578071 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 31983
I20260806 10:53:13.594391 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 31613
I20260806 10:53:13.595631 31677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:13.703152 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 31613
2026-08-06T10:53:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4108 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:53:13 dist-test-slave-49h3 krb5kdc[32137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:53:13 dist-test-slave-49h3 krb5kdc[32137](info): set up 2 sockets
Aug 06 10:53:13 dist-test-slave-49h3 krb5kdc[32137](info): commencing operation
krb5kdc: starting...
W20260806 10:53:14.741869 31592 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:53:14 dist-test-slave-49h3 krb5kdc[32137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:53:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:53:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:53:14.967502 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:38975
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:38975
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:15.311272 32150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:15.311622 32150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:15.311748 32150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:15.319482 32150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:53:15.319628 32150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:15.319685 32150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:15.319770 32150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:53:15.319829 32150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:53:15.330394 32150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:38975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:38975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:15.331867 32150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:15.333338 32150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:15.345973 32155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:15.346015 32158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:15.346056 32156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:15.346836 32150 server_base.cc:1034] running on GCE node
I20260806 10:53:15.347711 32150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:15.349370 32150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:15.350652 32150 hybrid_clock.cc:648] HybridClock initialized: now 1786013595350583 us; error 58 us; skew 500 ppm
I20260806 10:53:15.351071 32150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:15 dist-test-slave-49h3 krb5kdc[32137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013595, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:15.359184 32150 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:53:15.360916 32150 webserver.cc:460] Webserver started at http://127.30.218.62:38953/ using document root <none> and password file <none>
I20260806 10:53:15.361783 32150 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:15.361922 32150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:15.362259 32150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:15.365567 32150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "3e1be7ede50443afa87f6c99a22fdb74"
format_stamp: "Formatted at 2026-08-06 10:53:15 on dist-test-slave-49h3"
server_key: "d3962ac0e363b3d5cc13ecbf481342a5"
server_key_iv: "9c2f91ae056f5cc7e7399616bd174399"
server_key_version: "encryptionkey@0"
I20260806 10:53:15.366405 32150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "3e1be7ede50443afa87f6c99a22fdb74"
format_stamp: "Formatted at 2026-08-06 10:53:15 on dist-test-slave-49h3"
server_key: "d3962ac0e363b3d5cc13ecbf481342a5"
server_key_iv: "9c2f91ae056f5cc7e7399616bd174399"
server_key_version: "encryptionkey@0"
I20260806 10:53:15.371977 32150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 10:53:15.375571 32165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:15.376914 32150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:53:15.377082 32150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "3e1be7ede50443afa87f6c99a22fdb74"
format_stamp: "Formatted at 2026-08-06 10:53:15 on dist-test-slave-49h3"
server_key: "d3962ac0e363b3d5cc13ecbf481342a5"
server_key_iv: "9c2f91ae056f5cc7e7399616bd174399"
server_key_version: "encryptionkey@0"
I20260806 10:53:15.377305 32150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:15.403792 32150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:15.408512 32150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:15.408823 32150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:15.426781 32150 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:38975
I20260806 10:53:15.426789 32217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:38975 every 8 connection(s)
I20260806 10:53:15.428224 32150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:53:15.429915 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 32150
I20260806 10:53:15.430127 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:53:15.430801 31592 external_mini_cluster.cc:1428] Setting key f9bc00eac94999ffe639c6956239688f
I20260806 10:53:15.432394 32218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 10:53:15 dist-test-slave-49h3 krb5kdc[32137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:15.446767 32218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e1be7ede50443afa87f6c99a22fdb74: Bootstrap starting.
I20260806 10:53:15.451104 32218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e1be7ede50443afa87f6c99a22fdb74: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:15.452370 32218 log.cc:824] T 00000000000000000000000000000000 P 3e1be7ede50443afa87f6c99a22fdb74: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:15.455988 32218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e1be7ede50443afa87f6c99a22fdb74: No bootstrap required, opened a new log
I20260806 10:53:15.456135 32221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:15.432787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37254 (local address 127.30.218.62:38975)
0806 10:53:15.433608 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:15.433622 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:15.433674 (+ 52us) server_negotiation.cc:407] Connection header received
0806 10:53:15.435770 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:15.435802 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:15.436365 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:15.436878 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:15.438419 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:15.440512 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:15.441861 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:15.442917 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:15.443115 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:15.447336 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:15.447348 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:15.447370 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:15.447428 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:15.451347 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:15.452138 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:15.452153 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:15.452162 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:15.452389 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:15.452769 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:15.452774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:15.452777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:15.453487 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:15.453747 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:15.454323 (+ 576us) server_negotiation.cc:299] Negotiation successful
0806 10:53:15.454800 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":311,"threads_started":1}
I20260806 10:53:15.463816 32218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e1be7ede50443afa87f6c99a22fdb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1be7ede50443afa87f6c99a22fdb74" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 38975 } }
I20260806 10:53:15.464264 32218 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e1be7ede50443afa87f6c99a22fdb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:15.464345 32218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e1be7ede50443afa87f6c99a22fdb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e1be7ede50443afa87f6c99a22fdb74, State: Initialized, Role: FOLLOWER
I20260806 10:53:15.464897 32218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e1be7ede50443afa87f6c99a22fdb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1be7ede50443afa87f6c99a22fdb74" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 38975 } }
I20260806 10:53:15.465124 32218 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e1be7ede50443afa87f6c99a22fdb74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:53:15.465229 32218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e1be7ede50443afa87f6c99a22fdb74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:53:15.465360 32218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e1be7ede50443afa87f6c99a22fdb74 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:15.467785 32218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e1be7ede50443afa87f6c99a22fdb74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1be7ede50443afa87f6c99a22fdb74" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 38975 } }
I20260806 10:53:15.468318 32218 leader_election.cc:302] T 00000000000000000000000000000000 P 3e1be7ede50443afa87f6c99a22fdb74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e1be7ede50443afa87f6c99a22fdb74; no voters:
I20260806 10:53:15.468900 32218 leader_election.cc:288] T 00000000000000000000000000000000 P 3e1be7ede50443afa87f6c99a22fdb74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:53:15.469041 32223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e1be7ede50443afa87f6c99a22fdb74 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:15.469715 32223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e1be7ede50443afa87f6c99a22fdb74 [term 1 LEADER]: Becoming Leader. State: Replica: 3e1be7ede50443afa87f6c99a22fdb74, State: Running, Role: LEADER
I20260806 10:53:15.470302 32223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e1be7ede50443afa87f6c99a22fdb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1be7ede50443afa87f6c99a22fdb74" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 38975 } }
I20260806 10:53:15.471029 32218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e1be7ede50443afa87f6c99a22fdb74 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:53:15.473210 32225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e1be7ede50443afa87f6c99a22fdb74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e1be7ede50443afa87f6c99a22fdb74. Latest consensus state: current_term: 1 leader_uuid: "3e1be7ede50443afa87f6c99a22fdb74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1be7ede50443afa87f6c99a22fdb74" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 38975 } } }
I20260806 10:53:15.473749 32225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e1be7ede50443afa87f6c99a22fdb74 [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:15.473660 32224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e1be7ede50443afa87f6c99a22fdb74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e1be7ede50443afa87f6c99a22fdb74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1be7ede50443afa87f6c99a22fdb74" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 38975 } } }
I20260806 10:53:15.474445 32224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e1be7ede50443afa87f6c99a22fdb74 [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:15.478708 32229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:53:15.484226 32229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:53:15.495160 32229 catalog_manager.cc:1347] Generated new cluster ID: 2f57e5c4c3df4f73a21e8f80ae82f74e
I20260806 10:53:15.495306 32229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:53:15.514730 32229 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:53:15.516096 32229 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:53:15.522919 32229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e1be7ede50443afa87f6c99a22fdb74: Generated new TSK 0
I20260806 10:53:15.524047 32229 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
I20260806 10:53:15.597501 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:38975
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:15.975642 32246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:15.976001 32246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:15.976161 32246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:15.984071 32246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:15.984232 32246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:15.984395 32246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:53:15.995033 32246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:38975
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:15.996510 32246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:15.997987 32246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:16.012001 32251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:16.012097 32254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:16.012037 32252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:16.012966 32246 server_base.cc:1034] running on GCE node
I20260806 10:53:16.013659 32246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:16.014660 32246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:16.015874 32246 hybrid_clock.cc:648] HybridClock initialized: now 1786013596015819 us; error 46 us; skew 500 ppm
I20260806 10:53:16.016201 32246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:16 dist-test-slave-49h3 krb5kdc[32137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013596, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:16.019618 32246 init.cc:377] Logged in from keytab as kudu/127.30.218.1@KRBTEST.COM (short username kudu)
I20260806 10:53:16.021219 32246 webserver.cc:460] Webserver started at http://127.30.218.1:34739/ using document root <none> and password file <none>
I20260806 10:53:16.022027 32246 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:16.022148 32246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:16.022449 32246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:16.025701 32246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "3df01539e2c443caac08c6955e09b10b"
format_stamp: "Formatted at 2026-08-06 10:53:16 on dist-test-slave-49h3"
server_key: "15b7c958d4ec5da4d67439a0aab5ffeb"
server_key_iv: "63e83b51d49bda72d020d6a85d496018"
server_key_version: "encryptionkey@0"
I20260806 10:53:16.026609 32246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "3df01539e2c443caac08c6955e09b10b"
format_stamp: "Formatted at 2026-08-06 10:53:16 on dist-test-slave-49h3"
server_key: "15b7c958d4ec5da4d67439a0aab5ffeb"
server_key_iv: "63e83b51d49bda72d020d6a85d496018"
server_key_version: "encryptionkey@0"
I20260806 10:53:16.032383 32246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 10:53:16.036367 32263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:16.037945 32246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:53:16.038141 32246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "3df01539e2c443caac08c6955e09b10b"
format_stamp: "Formatted at 2026-08-06 10:53:16 on dist-test-slave-49h3"
server_key: "15b7c958d4ec5da4d67439a0aab5ffeb"
server_key_iv: "63e83b51d49bda72d020d6a85d496018"
server_key_version: "encryptionkey@0"
I20260806 10:53:16.038355 32246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:16.066360 32246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:16.071092 32246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:16.071398 32246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:16.072525 32246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:53:16.074152 32246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:16.074246 32246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:16.074376 32246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:16.074430 32246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:16.106683 32246 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:34693
I20260806 10:53:16.106721 32377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:34693 every 8 connection(s)
I20260806 10:53:16.108096 32246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:53:16.115365 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 32246
I20260806 10:53:16.115592 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:53:16.116276 31592 external_mini_cluster.cc:1428] Setting key 3f9de372fec6778efc5e138a809fd5c1
Aug 06 10:53:16 dist-test-slave-49h3 krb5kdc[32137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013596, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:16.130851 32380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:16.110810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:44569 (local address 127.30.218.62:38975)
0806 10:53:16.111313 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:16.111318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:16.112222 (+ 904us) server_negotiation.cc:407] Connection header received
0806 10:53:16.114321 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:16.114327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:16.114419 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:16.114526 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:16.116392 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:16.117213 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:16.119457 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:16.120036 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:16.120202 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:16.124227 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:16.124234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:16.124238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:16.124285 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:16.127267 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:16.128318 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:16.128322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:16.128324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:16.128455 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:16.128953 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:16.128956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:16.128958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:16.129247 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:16.129370 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:16.130360 (+ 990us) server_negotiation.cc:299] Negotiation successful
0806 10:53:16.130569 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":255,"threads_started":1}
I20260806 10:53:16.132396 32381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:16.111265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:38975 (local address 127.30.218.1:44569)
0806 10:53:16.112022 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:16.112072 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:16.113997 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:16.114850 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:16.114859 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:16.115438 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:16.116178 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:16.116199 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:16.117420 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:16.117426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:16.119240 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:16.119263 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:16.120342 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:16.120349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:16.120595 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:16.121351 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:16.121378 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:16.123924 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:16.127469 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:16.127481 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:16.127502 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:16.128189 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:16.128558 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:16.128563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:16.128567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:16.128849 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:16.129382 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:16.129391 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:16.129920 (+ 529us) client_negotiation.cc:769] Sending connection context
0806 10:53:16.130256 (+ 336us) client_negotiation.cc:240] Negotiation successful
0806 10:53:16.130936 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":289,"threads_started":1}
I20260806 10:53:16.134290 32378 heartbeater.cc:347] Connected to a master server at 127.30.218.62:38975
I20260806 10:53:16.134709 32378 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:16.135811 32378 heartbeater.cc:511] Master 127.30.218.62:38975 requested a full tablet report, sending...
I20260806 10:53:16.138550 32183 ts_manager.cc:194] Registered new tserver with Master: 3df01539e2c443caac08c6955e09b10b (127.30.218.1:34693)
WARNING: no policy specified for kudu/127.30.218.2@KRBTEST.COM; defaulting to no policy
I20260806 10:53:16.140213 32183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.1@KRBTEST.COM'} at 127.30.218.1:44569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
I20260806 10:53:16.169148 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:38975
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:16.514070 32384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:16.514384 32384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:16.514513 32384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:16.521983 32384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:16.522121 32384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:16.522286 32384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:53:16.532541 32384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:38975
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:16.534288 32384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:16.535791 32384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:16.548777 32389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:16.548787 32390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:16.549031 32392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:16.549759 32384 server_base.cc:1034] running on GCE node
I20260806 10:53:16.550457 32384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:16.551504 32384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:16.552737 32384 hybrid_clock.cc:648] HybridClock initialized: now 1786013596552663 us; error 63 us; skew 500 ppm
I20260806 10:53:16.553077 32384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:16 dist-test-slave-49h3 krb5kdc[32137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013596, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:16.556495 32384 init.cc:377] Logged in from keytab as kudu/127.30.218.2@KRBTEST.COM (short username kudu)
I20260806 10:53:16.558269 32384 webserver.cc:460] Webserver started at http://127.30.218.2:34409/ using document root <none> and password file <none>
I20260806 10:53:16.559034 32384 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:16.559168 32384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:16.559494 32384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:16.562984 32384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "dd8d0c1665d644e5a390240bf8c47fe8"
format_stamp: "Formatted at 2026-08-06 10:53:16 on dist-test-slave-49h3"
server_key: "cc9cdb38f59e0076d54607be7309b98e"
server_key_iv: "e4f889a0a1e49ed3cb33aef2745f5b88"
server_key_version: "encryptionkey@0"
I20260806 10:53:16.563870 32384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "dd8d0c1665d644e5a390240bf8c47fe8"
format_stamp: "Formatted at 2026-08-06 10:53:16 on dist-test-slave-49h3"
server_key: "cc9cdb38f59e0076d54607be7309b98e"
server_key_iv: "e4f889a0a1e49ed3cb33aef2745f5b88"
server_key_version: "encryptionkey@0"
I20260806 10:53:16.569831 32384 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260806 10:53:16.573921 32399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:16.575539 32384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:53:16.575763 32384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "dd8d0c1665d644e5a390240bf8c47fe8"
format_stamp: "Formatted at 2026-08-06 10:53:16 on dist-test-slave-49h3"
server_key: "cc9cdb38f59e0076d54607be7309b98e"
server_key_iv: "e4f889a0a1e49ed3cb33aef2745f5b88"
server_key_version: "encryptionkey@0"
I20260806 10:53:16.576011 32384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:16.592499 32384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:16.597081 32384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:16.597417 32384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:16.598562 32384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:53:16.600106 32384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:16.600203 32384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:16.600337 32384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:16.600399 32384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:16.633222 32384 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:42477
I20260806 10:53:16.633249 32513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:42477 every 8 connection(s)
I20260806 10:53:16.634655 32384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:53:16.643992 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 32384
I20260806 10:53:16.644186 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:53:16.644637 31592 external_mini_cluster.cc:1428] Setting key e6b6f112dfb42a5cff6c2d94592393a4
Aug 06 10:53:16 dist-test-slave-49h3 krb5kdc[32137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013596, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:16.657295 32516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:16.637367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:47031 (local address 127.30.218.62:38975)
0806 10:53:16.637857 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:16.637862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:16.638844 (+ 982us) server_negotiation.cc:407] Connection header received
0806 10:53:16.640732 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:16.640738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:16.640822 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:16.640922 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:16.642606 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:16.643362 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:16.645080 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:16.645704 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:16.645826 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:16.650171 (+ 4345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:16.650178 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:16.650183 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:16.650232 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:16.653721 (+ 3489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:16.654641 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:16.654647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:16.654649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:16.654853 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:16.655404 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:16.655407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:16.655409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:16.655713 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:16.655845 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:16.656770 (+ 925us) server_negotiation.cc:299] Negotiation successful
0806 10:53:16.656971 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":257,"threads_started":1}
I20260806 10:53:16.658762 32517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:16.637947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:38975 (local address 127.30.218.2:47031)
0806 10:53:16.638670 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:16.638713 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:16.640447 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:16.641168 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:16.641177 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:16.641777 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:16.642431 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:16.642454 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:16.643521 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:16.643527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:16.644932 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:16.644946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:16.645939 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:16.645947 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:16.646207 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:16.646906 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:16.646932 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:16.649854 (+ 2922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:16.653926 (+ 4072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:16.653935 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:16.653952 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:16.654480 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:16.654957 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:16.654961 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:16.654964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:16.655285 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:16.655846 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:16.655855 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:16.656377 (+ 522us) client_negotiation.cc:769] Sending connection context
0806 10:53:16.656680 (+ 303us) client_negotiation.cc:240] Negotiation successful
0806 10:53:16.657296 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":276,"threads_started":1}
I20260806 10:53:16.660377 32514 heartbeater.cc:347] Connected to a master server at 127.30.218.62:38975
I20260806 10:53:16.660751 32514 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:16.661790 32514 heartbeater.cc:511] Master 127.30.218.62:38975 requested a full tablet report, sending...
I20260806 10:53:16.663596 32183 ts_manager.cc:194] Registered new tserver with Master: dd8d0c1665d644e5a390240bf8c47fe8 (127.30.218.2:42477)
I20260806 10:53:16.664510 32183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.2@KRBTEST.COM'} at 127.30.218.2:47031
WARNING: no policy specified for kudu/127.30.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
I20260806 10:53:16.702128 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:38975
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:17.049307 32520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:17.049710 32520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:17.049867 32520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:17.057415 32520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:17.057631 32520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:17.057819 32520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:53:17.068321 32520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:38975
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:17.069796 32520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:17.071164 32520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:17.085084 32528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:17.085182 32525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:17.085111 32526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:17.091284 32520 server_base.cc:1034] running on GCE node
I20260806 10:53:17.092072 32520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:17.093185 32520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:17.094410 32520 hybrid_clock.cc:648] HybridClock initialized: now 1786013597094337 us; error 69 us; skew 500 ppm
I20260806 10:53:17.094759 32520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:17 dist-test-slave-49h3 krb5kdc[32137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013597, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:17.098213 32520 init.cc:377] Logged in from keytab as kudu/127.30.218.3@KRBTEST.COM (short username kudu)
I20260806 10:53:17.099887 32520 webserver.cc:460] Webserver started at http://127.30.218.3:36159/ using document root <none> and password file <none>
I20260806 10:53:17.100641 32520 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:17.100752 32520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:17.101045 32520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:17.104210 32520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "43ffcec2bd3b45bf892a207db5dbb012"
format_stamp: "Formatted at 2026-08-06 10:53:17 on dist-test-slave-49h3"
server_key: "4b95a020113f0020c4d284d0368ddeba"
server_key_iv: "ed1fc50a9ad858d23f420f79754ff9f5"
server_key_version: "encryptionkey@0"
I20260806 10:53:17.105019 32520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "43ffcec2bd3b45bf892a207db5dbb012"
format_stamp: "Formatted at 2026-08-06 10:53:17 on dist-test-slave-49h3"
server_key: "4b95a020113f0020c4d284d0368ddeba"
server_key_iv: "ed1fc50a9ad858d23f420f79754ff9f5"
server_key_version: "encryptionkey@0"
I20260806 10:53:17.110617 32520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260806 10:53:17.114208 32535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:17.115548 32520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:53:17.115702 32520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "43ffcec2bd3b45bf892a207db5dbb012"
format_stamp: "Formatted at 2026-08-06 10:53:17 on dist-test-slave-49h3"
server_key: "4b95a020113f0020c4d284d0368ddeba"
server_key_iv: "ed1fc50a9ad858d23f420f79754ff9f5"
server_key_version: "encryptionkey@0"
I20260806 10:53:17.115886 32520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:17.135119 32520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:17.139545 32520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:17.139797 32520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:17.140743 32520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:53:17.142163 32520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:17.142237 32520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:17.142331 32520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:17.142373 32520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:17.144078 32378 heartbeater.cc:503] Master 127.30.218.62:38975 was elected leader, sending a full tablet report...
I20260806 10:53:17.174134 32520 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:35061
I20260806 10:53:17.174300 32651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:35061 every 8 connection(s)
I20260806 10:53:17.175647 32520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:53:17.177829 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 32520
I20260806 10:53:17.178068 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:53:17.178683 31592 external_mini_cluster.cc:1428] Setting key 61bf8a0a3b152a0aeef8aefa1ca7f490
Aug 06 10:53:17 dist-test-slave-49h3 krb5kdc[32137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013597, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:17.198849 32654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:17.178538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:53937 (local address 127.30.218.62:38975)
0806 10:53:17.179003 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:17.179008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:17.180401 (+ 1393us) server_negotiation.cc:407] Connection header received
0806 10:53:17.182897 (+ 2496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:17.182902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:17.183006 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:17.183101 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:17.184827 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.185670 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.187361 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.187794 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.187911 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:17.191797 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:17.191803 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:17.191806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:17.191850 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:17.195320 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:17.196232 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:17.196239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:17.196243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:17.196388 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:17.196854 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:17.196859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:17.196861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:17.197159 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:17.197313 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:17.198163 (+ 850us) server_negotiation.cc:299] Negotiation successful
0806 10:53:17.198425 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":224,"threads_started":1}
I20260806 10:53:17.199949 32655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:17.179291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:38975 (local address 127.30.218.3:53937)
0806 10:53:17.180220 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:17.180275 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:17.182578 (+ 2303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:17.183395 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:17.183405 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:17.183994 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:17.184636 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:17.184657 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.185839 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.185843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:17.187203 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:17.187218 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.188052 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.188060 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:17.188312 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:17.189025 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:17.189057 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:17.191505 (+ 2448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:17.195520 (+ 4015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:17.195530 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:17.195550 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:17.196102 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:17.196509 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:17.196514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:17.196516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:17.196736 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:17.197299 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:17.197309 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:17.197752 (+ 443us) client_negotiation.cc:769] Sending connection context
0806 10:53:17.198052 (+ 300us) client_negotiation.cc:240] Negotiation successful
0806 10:53:17.198682 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":317,"threads_started":1}
I20260806 10:53:17.201550 32652 heartbeater.cc:347] Connected to a master server at 127.30.218.62:38975
I20260806 10:53:17.201881 32652 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:17.202685 32652 heartbeater.cc:511] Master 127.30.218.62:38975 requested a full tablet report, sending...
I20260806 10:53:17.204391 32183 ts_manager.cc:194] Registered new tserver with Master: 43ffcec2bd3b45bf892a207db5dbb012 (127.30.218.3:35061)
I20260806 10:53:17.205257 32183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.3@KRBTEST.COM'} at 127.30.218.3:53937
I20260806 10:53:17.208178 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:53:17.228783 32654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:17.215161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47872 (local address 127.30.218.62:38975)
0806 10:53:17.215348 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:17.215352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:17.215875 (+ 523us) server_negotiation.cc:407] Connection header received
0806 10:53:17.216041 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:17.216045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:17.216137 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:17.216243 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:17.217882 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.218694 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.220165 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.220645 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.220754 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:17.223364 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:17.223370 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:17.223373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:17.223416 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:17.226237 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:17.226991 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:17.226994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:17.226996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:17.227157 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:17.227489 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:17.227492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:17.227494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:17.227848 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:17.228078 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:17.228332 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 10:53:17.228532 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 10:53:17.236181 32183 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:53:17.243656 32183 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:53:17.287209 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:17.259853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:42477 (local address 127.0.0.1:38704)
0806 10:53:17.274920 (+ 15067us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:17.274993 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:17.275280 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:17.277084 (+ 1804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:17.277094 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:17.277140 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:53:17.277620 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:17.277637 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.281894 (+ 4257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.281900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:17.283884 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:17.283898 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.286051 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.286059 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:17.286286 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:17.286336 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 10:53:17.286474 (+ 138us) client_negotiation.cc:240] Negotiation successful
0806 10:53:17.286718 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14666,"mutex_wait_us":26,"thread_start_us":16073,"threads_started":1}
I20260806 10:53:17.287358 32668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:17.260257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:35061 (local address 127.0.0.1:50366)
0806 10:53:17.278170 (+ 17913us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:17.278192 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:17.278442 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:17.279528 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:17.279533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:17.279562 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:53:17.279981 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:17.279994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.281969 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.281972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:17.284524 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:17.284542 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.286527 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.286533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:17.286710 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:17.286740 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 10:53:17.286835 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 10:53:17.286972 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17769,"mutex_wait_us":79,"thread_start_us":18129,"threads_started":1}
I20260806 10:53:17.287882 32670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:17.261466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50366 (local address 127.30.218.3:35061)
0806 10:53:17.271785 (+ 10319us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:17.271796 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:17.278318 (+ 6522us) server_negotiation.cc:407] Connection header received
0806 10:53:17.278676 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:17.278684 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:17.279010 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:17.279198 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:53:17.280207 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.281817 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.284759 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.286308 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.286446 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:17.286676 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:17.286922 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 10:53:17.287099 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10059,"thread_start_us":9417,"threads_started":1}
I20260806 10:53:17.287940 32669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:17.261038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52950 (local address 127.30.218.1:34693)
0806 10:53:17.277685 (+ 16647us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:17.277694 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:17.278778 (+ 1084us) server_negotiation.cc:407] Connection header received
0806 10:53:17.278914 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:17.278919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:17.279120 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:17.279283 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:53:17.279953 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.282157 (+ 2204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.284770 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.286317 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.286446 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:17.286680 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:17.287465 (+ 785us) server_negotiation.cc:299] Negotiation successful
0806 10:53:17.287640 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16446,"thread_start_us":17055,"threads_started":1}
I20260806 10:53:17.288167 32666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:17.260044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38704 (local address 127.30.218.2:42477)
0806 10:53:17.275979 (+ 15935us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:17.275989 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:17.276022 (+ 33us) server_negotiation.cc:407] Connection header received
0806 10:53:17.276132 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:17.276140 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:17.276434 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:17.276639 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:53:17.277900 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.281729 (+ 3829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.284355 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.285813 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.285948 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:17.286216 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:17.287731 (+ 1515us) server_negotiation.cc:299] Negotiation successful
0806 10:53:17.287887 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15762,"thread_start_us":15636,"threads_started":1}
I20260806 10:53:17.288472 32667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:17.260608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:34693 (local address 127.0.0.1:52950)
0806 10:53:17.278647 (+ 18039us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:17.278662 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:17.278781 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:17.279290 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:17.279296 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:17.279373 (+ 77us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:53:17.279794 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:17.279805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.282288 (+ 2483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.282294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:17.284590 (+ 2296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:17.284600 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.286827 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.286834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:17.287019 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:17.287057 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 10:53:17.287162 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 10:53:17.287320 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17941,"mutex_wait_us":13,"thread_start_us":17272,"threads_started":1}
I20260806 10:53:17.292135 32311 tablet_service.cc:1467] Processing CreateTablet for tablet 8415e1eedbe64c57a0b2c5698aa8fe14 (DEFAULT_TABLE table=test-table [id=b61363a54e514c53907ac9a64bbf3973]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:17.292145 32584 tablet_service.cc:1467] Processing CreateTablet for tablet 8415e1eedbe64c57a0b2c5698aa8fe14 (DEFAULT_TABLE table=test-table [id=b61363a54e514c53907ac9a64bbf3973]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:17.292145 32448 tablet_service.cc:1467] Processing CreateTablet for tablet 8415e1eedbe64c57a0b2c5698aa8fe14 (DEFAULT_TABLE table=test-table [id=b61363a54e514c53907ac9a64bbf3973]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:17.293594 32311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8415e1eedbe64c57a0b2c5698aa8fe14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:17.293615 32584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8415e1eedbe64c57a0b2c5698aa8fe14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:17.293603 32448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8415e1eedbe64c57a0b2c5698aa8fe14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:17.305724 32671 tablet_bootstrap.cc:492] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b: Bootstrap starting.
I20260806 10:53:17.310021 32671 tablet_bootstrap.cc:654] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:17.310398 32673 tablet_bootstrap.cc:492] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 43ffcec2bd3b45bf892a207db5dbb012: Bootstrap starting.
I20260806 10:53:17.311661 32671 log.cc:824] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:17.313300 32672 tablet_bootstrap.cc:492] T 8415e1eedbe64c57a0b2c5698aa8fe14 P dd8d0c1665d644e5a390240bf8c47fe8: Bootstrap starting.
I20260806 10:53:17.314514 32673 tablet_bootstrap.cc:654] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 43ffcec2bd3b45bf892a207db5dbb012: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:17.315235 32671 tablet_bootstrap.cc:492] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b: No bootstrap required, opened a new log
I20260806 10:53:17.315506 32671 ts_tablet_manager.cc:1397] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260806 10:53:17.315850 32673 log.cc:824] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 43ffcec2bd3b45bf892a207db5dbb012: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:17.317047 32672 tablet_bootstrap.cc:654] T 8415e1eedbe64c57a0b2c5698aa8fe14 P dd8d0c1665d644e5a390240bf8c47fe8: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:17.318328 32672 log.cc:824] T 8415e1eedbe64c57a0b2c5698aa8fe14 P dd8d0c1665d644e5a390240bf8c47fe8: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:17.319693 32673 tablet_bootstrap.cc:492] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 43ffcec2bd3b45bf892a207db5dbb012: No bootstrap required, opened a new log
I20260806 10:53:17.319994 32673 ts_tablet_manager.cc:1397] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 43ffcec2bd3b45bf892a207db5dbb012: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260806 10:53:17.321185 32672 tablet_bootstrap.cc:492] T 8415e1eedbe64c57a0b2c5698aa8fe14 P dd8d0c1665d644e5a390240bf8c47fe8: No bootstrap required, opened a new log
I20260806 10:53:17.321440 32672 ts_tablet_manager.cc:1397] T 8415e1eedbe64c57a0b2c5698aa8fe14 P dd8d0c1665d644e5a390240bf8c47fe8: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260806 10:53:17.326107 32671 raft_consensus.cc:348] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8d0c1665d644e5a390240bf8c47fe8" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42477 } } peers { permanent_uuid: "43ffcec2bd3b45bf892a207db5dbb012" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35061 } } peers { permanent_uuid: "3df01539e2c443caac08c6955e09b10b" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34693 } }
I20260806 10:53:17.326637 32671 raft_consensus.cc:374] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:17.326745 32671 raft_consensus.cc:729] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3df01539e2c443caac08c6955e09b10b, State: Initialized, Role: FOLLOWER
I20260806 10:53:17.327445 32671 consensus_queue.cc:260] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8d0c1665d644e5a390240bf8c47fe8" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42477 } } peers { permanent_uuid: "43ffcec2bd3b45bf892a207db5dbb012" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35061 } } peers { permanent_uuid: "3df01539e2c443caac08c6955e09b10b" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34693 } }
I20260806 10:53:17.329027 32672 raft_consensus.cc:348] T 8415e1eedbe64c57a0b2c5698aa8fe14 P dd8d0c1665d644e5a390240bf8c47fe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8d0c1665d644e5a390240bf8c47fe8" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42477 } } peers { permanent_uuid: "43ffcec2bd3b45bf892a207db5dbb012" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35061 } } peers { permanent_uuid: "3df01539e2c443caac08c6955e09b10b" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34693 } }
I20260806 10:53:17.329578 32672 raft_consensus.cc:374] T 8415e1eedbe64c57a0b2c5698aa8fe14 P dd8d0c1665d644e5a390240bf8c47fe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:17.329689 32672 raft_consensus.cc:729] T 8415e1eedbe64c57a0b2c5698aa8fe14 P dd8d0c1665d644e5a390240bf8c47fe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8d0c1665d644e5a390240bf8c47fe8, State: Initialized, Role: FOLLOWER
I20260806 10:53:17.330699 32673 raft_consensus.cc:348] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 43ffcec2bd3b45bf892a207db5dbb012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8d0c1665d644e5a390240bf8c47fe8" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42477 } } peers { permanent_uuid: "43ffcec2bd3b45bf892a207db5dbb012" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35061 } } peers { permanent_uuid: "3df01539e2c443caac08c6955e09b10b" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34693 } }
I20260806 10:53:17.331080 32671 ts_tablet_manager.cc:1428] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260806 10:53:17.331187 32673 raft_consensus.cc:374] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 43ffcec2bd3b45bf892a207db5dbb012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:17.331301 32673 raft_consensus.cc:729] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 43ffcec2bd3b45bf892a207db5dbb012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ffcec2bd3b45bf892a207db5dbb012, State: Initialized, Role: FOLLOWER
I20260806 10:53:17.331597 32672 consensus_queue.cc:260] T 8415e1eedbe64c57a0b2c5698aa8fe14 P dd8d0c1665d644e5a390240bf8c47fe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8d0c1665d644e5a390240bf8c47fe8" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42477 } } peers { permanent_uuid: "43ffcec2bd3b45bf892a207db5dbb012" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35061 } } peers { permanent_uuid: "3df01539e2c443caac08c6955e09b10b" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34693 } }
I20260806 10:53:17.331877 32673 consensus_queue.cc:260] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 43ffcec2bd3b45bf892a207db5dbb012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8d0c1665d644e5a390240bf8c47fe8" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42477 } } peers { permanent_uuid: "43ffcec2bd3b45bf892a207db5dbb012" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35061 } } peers { permanent_uuid: "3df01539e2c443caac08c6955e09b10b" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34693 } }
I20260806 10:53:17.332934 32514 heartbeater.cc:503] Master 127.30.218.62:38975 was elected leader, sending a full tablet report...
I20260806 10:53:17.333621 32672 ts_tablet_manager.cc:1428] T 8415e1eedbe64c57a0b2c5698aa8fe14 P dd8d0c1665d644e5a390240bf8c47fe8: Time spent starting tablet: real 0.012s user 0.003s sys 0.008s
I20260806 10:53:17.334487 32652 heartbeater.cc:503] Master 127.30.218.62:38975 was elected leader, sending a full tablet report...
I20260806 10:53:17.335397 32673 ts_tablet_manager.cc:1428] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 43ffcec2bd3b45bf892a207db5dbb012: Time spent starting tablet: real 0.015s user 0.007s sys 0.009s
W20260806 10:53:17.360561 32379 tablet.cc:2369] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:53:17.386837 32515 tablet.cc:2369] T 8415e1eedbe64c57a0b2c5698aa8fe14 P dd8d0c1665d644e5a390240bf8c47fe8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:53:17.395146 32677 raft_consensus.cc:482] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:53:17.395485 32677 raft_consensus.cc:504] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8d0c1665d644e5a390240bf8c47fe8" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42477 } } peers { permanent_uuid: "43ffcec2bd3b45bf892a207db5dbb012" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35061 } } peers { permanent_uuid: "3df01539e2c443caac08c6955e09b10b" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34693 } }
I20260806 10:53:17.397374 32677 leader_election.cc:288] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd8d0c1665d644e5a390240bf8c47fe8 (127.30.218.2:42477), 43ffcec2bd3b45bf892a207db5dbb012 (127.30.218.3:35061)
I20260806 10:53:17.406464 32666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:17.398190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:54969 (local address 127.30.218.2:42477)
0806 10:53:17.398418 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:17.398425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:17.398728 (+ 303us) server_negotiation.cc:407] Connection header received
0806 10:53:17.398949 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:17.398955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:17.399042 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:17.399153 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:53:17.399847 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.400983 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.404056 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.405319 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.405517 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:17.405598 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:17.406043 (+ 445us) server_negotiation.cc:299] Negotiation successful
0806 10:53:17.406190 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 10:53:17.406464 32670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:17.397992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:41041 (local address 127.30.218.3:35061)
0806 10:53:17.398397 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:17.398403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:17.398960 (+ 557us) server_negotiation.cc:407] Connection header received
0806 10:53:17.399093 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:17.399099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:17.399176 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:17.399263 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:53:17.399967 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.401067 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.404045 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.405319 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.405518 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:17.405598 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:17.406046 (+ 448us) server_negotiation.cc:299] Negotiation successful
0806 10:53:17.406190 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239}
I20260806 10:53:17.406736 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:17.397813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:42477 (local address 127.30.218.1:54969)
0806 10:53:17.398581 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:17.398603 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:17.398799 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:17.399174 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:17.399178 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:17.399199 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:53:17.399645 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:17.399663 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.401183 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.401189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:17.403841 (+ 2652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:17.403860 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.405611 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.405619 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:17.405851 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:17.405888 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 10:53:17.405992 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 10:53:17.406252 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":191,"thread_start_us":410,"threads_started":1}
I20260806 10:53:17.407065 32681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:17.397932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:35061 (local address 127.30.218.1:41041)
0806 10:53:17.398861 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:17.398877 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:17.398982 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:17.399302 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:17.399306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:17.399346 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:53:17.399800 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:17.399812 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.401207 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.401211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:17.403838 (+ 2627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:17.403860 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.405635 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.405641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:17.405846 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:17.405899 (+ 53us) client_negotiation.cc:769] Sending connection context
0806 10:53:17.405988 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 10:53:17.406252 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":212,"thread_start_us":629,"threads_started":1}
I20260806 10:53:17.407785 32468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8415e1eedbe64c57a0b2c5698aa8fe14" candidate_uuid: "3df01539e2c443caac08c6955e09b10b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8d0c1665d644e5a390240bf8c47fe8" is_pre_election: true
I20260806 10:53:17.407984 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8415e1eedbe64c57a0b2c5698aa8fe14" candidate_uuid: "3df01539e2c443caac08c6955e09b10b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43ffcec2bd3b45bf892a207db5dbb012" is_pre_election: true
I20260806 10:53:17.408321 32468 raft_consensus.cc:2454] T 8415e1eedbe64c57a0b2c5698aa8fe14 P dd8d0c1665d644e5a390240bf8c47fe8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3df01539e2c443caac08c6955e09b10b in term 0.
I20260806 10:53:17.408506 32604 raft_consensus.cc:2454] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 43ffcec2bd3b45bf892a207db5dbb012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3df01539e2c443caac08c6955e09b10b in term 0.
I20260806 10:53:17.409047 32264 leader_election.cc:302] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3df01539e2c443caac08c6955e09b10b, dd8d0c1665d644e5a390240bf8c47fe8; no voters:
I20260806 10:53:17.409561 32677 raft_consensus.cc:2781] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:53:17.409732 32677 raft_consensus.cc:482] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:53:17.409844 32677 raft_consensus.cc:3037] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:17.412328 32677 raft_consensus.cc:504] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8d0c1665d644e5a390240bf8c47fe8" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42477 } } peers { permanent_uuid: "43ffcec2bd3b45bf892a207db5dbb012" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35061 } } peers { permanent_uuid: "3df01539e2c443caac08c6955e09b10b" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34693 } }
I20260806 10:53:17.412995 32677 leader_election.cc:288] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b [CANDIDATE]: Term 1 election: Requested vote from peers dd8d0c1665d644e5a390240bf8c47fe8 (127.30.218.2:42477), 43ffcec2bd3b45bf892a207db5dbb012 (127.30.218.3:35061)
I20260806 10:53:17.413599 32468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8415e1eedbe64c57a0b2c5698aa8fe14" candidate_uuid: "3df01539e2c443caac08c6955e09b10b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8d0c1665d644e5a390240bf8c47fe8"
I20260806 10:53:17.413666 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8415e1eedbe64c57a0b2c5698aa8fe14" candidate_uuid: "3df01539e2c443caac08c6955e09b10b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43ffcec2bd3b45bf892a207db5dbb012"
I20260806 10:53:17.413826 32468 raft_consensus.cc:3037] T 8415e1eedbe64c57a0b2c5698aa8fe14 P dd8d0c1665d644e5a390240bf8c47fe8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:17.413887 32604 raft_consensus.cc:3037] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 43ffcec2bd3b45bf892a207db5dbb012 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:17.416378 32604 raft_consensus.cc:2454] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 43ffcec2bd3b45bf892a207db5dbb012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3df01539e2c443caac08c6955e09b10b in term 1.
I20260806 10:53:17.416447 32468 raft_consensus.cc:2454] T 8415e1eedbe64c57a0b2c5698aa8fe14 P dd8d0c1665d644e5a390240bf8c47fe8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3df01539e2c443caac08c6955e09b10b in term 1.
I20260806 10:53:17.416958 32267 leader_election.cc:302] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3df01539e2c443caac08c6955e09b10b, 43ffcec2bd3b45bf892a207db5dbb012, dd8d0c1665d644e5a390240bf8c47fe8; no voters:
I20260806 10:53:17.417323 32677 raft_consensus.cc:2781] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:17.417827 32677 raft_consensus.cc:686] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b [term 1 LEADER]: Becoming Leader. State: Replica: 3df01539e2c443caac08c6955e09b10b, State: Running, Role: LEADER
I20260806 10:53:17.418403 32677 consensus_queue.cc:237] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8d0c1665d644e5a390240bf8c47fe8" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42477 } } peers { permanent_uuid: "43ffcec2bd3b45bf892a207db5dbb012" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35061 } } peers { permanent_uuid: "3df01539e2c443caac08c6955e09b10b" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34693 } }
I20260806 10:53:17.423314 32181 catalog_manager.cc:5495] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 3df01539e2c443caac08c6955e09b10b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3df01539e2c443caac08c6955e09b10b (127.30.218.1). New cstate: current_term: 1 leader_uuid: "3df01539e2c443caac08c6955e09b10b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8d0c1665d644e5a390240bf8c47fe8" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43ffcec2bd3b45bf892a207db5dbb012" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3df01539e2c443caac08c6955e09b10b" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34693 } health_report { overall_health: HEALTHY } } }
W20260806 10:53:17.427379 32653 tablet.cc:2369] T 8415e1eedbe64c57a0b2c5698aa8fe14 P 43ffcec2bd3b45bf892a207db5dbb012: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 10:53:17 dist-test-slave-49h3 krb5kdc[32137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.1@KRBTEST.COM
I20260806 10:53:17.452171 32669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:17.436103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.30.218.1:34693)
0806 10:53:17.436341 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:17.436348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:17.436732 (+ 384us) server_negotiation.cc:407] Connection header received
0806 10:53:17.436894 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:17.436900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:17.436988 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:17.437117 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:17.438829 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.439993 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.441340 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.442110 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.442263 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:17.446674 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:17.446693 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:17.446712 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:17.446756 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:17.449631 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:17.450379 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:17.450387 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:17.450390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:17.450561 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:17.450868 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:17.450872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:17.450874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:17.451261 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:17.451493 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:17.451705 (+ 212us) server_negotiation.cc:299] Negotiation successful
0806 10:53:17.451900 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Aug 06 10:53:17 dist-test-slave-49h3 krb5kdc[32137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 10:53:17 dist-test-slave-49h3 krb5kdc[32137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.218.1@KRBTEST.COM
I20260806 10:53:17.508394 32669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:17.493087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52958 (local address 127.30.218.1:34693)
0806 10:53:17.493284 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:17.493290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:17.493742 (+ 452us) server_negotiation.cc:407] Connection header received
0806 10:53:17.493930 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:17.493935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:17.494006 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:17.494103 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:17.495593 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.496342 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.497745 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.498211 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.498335 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:17.502487 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:17.502493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:17.502496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:17.502542 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:17.505345 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:17.506226 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:17.506230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:17.506233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:17.506387 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:17.506857 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:17.506861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:17.506863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:17.507183 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:17.507494 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:17.507787 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 10:53:17.508044 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Aug 06 10:53:17 dist-test-slave-49h3 krb5kdc[32137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 10:53:17 dist-test-slave-49h3 krb5kdc[32137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.1@KRBTEST.COM
I20260806 10:53:17.564580 32669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:17.549187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52968 (local address 127.30.218.1:34693)
0806 10:53:17.549415 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:17.549422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:17.549777 (+ 355us) server_negotiation.cc:407] Connection header received
0806 10:53:17.549967 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:17.549970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:17.550060 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:17.550151 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:17.551695 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.552476 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.553848 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.554462 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.554609 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:17.559166 (+ 4557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:17.559173 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:17.559176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:17.559224 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:17.561905 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:17.562803 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:17.562809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:17.562811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:17.562974 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:17.563324 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:17.563327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:17.563329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:17.563636 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:17.563840 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:17.564096 (+ 256us) server_negotiation.cc:299] Negotiation successful
0806 10:53:17.564293 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260806 10:53:17.566707 32291 tablet_service.cc:2908] Rejecting scan request for tablet 8415e1eedbe64c57a0b2c5698aa8fe14: Uninitialized: clean time has not yet been initialized
I20260806 10:53:17.570051 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 32246
I20260806 10:53:17.583820 32669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:17.571087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52978 (local address 127.30.218.1:34693)
0806 10:53:17.571261 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:17.571265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:17.571671 (+ 406us) server_negotiation.cc:407] Connection header received
0806 10:53:17.571854 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:17.571860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:17.571930 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:17.572031 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:17.573273 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.574080 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.575413 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.575899 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.576055 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:17.578488 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:17.578493 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:17.578497 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:17.578536 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:17.581149 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:17.581990 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:17.581995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:17.581997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:17.582148 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:17.582510 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:17.582514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:17.582516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:17.582845 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:17.583050 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:17.583285 (+ 235us) server_negotiation.cc:299] Negotiation successful
0806 10:53:17.583511 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 10:53:17.584565 32372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:17.679672 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 32246
I20260806 10:53:17.699932 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 32384
Aug 06 10:53:17 dist-test-slave-49h3 krb5kdc[32137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.2@KRBTEST.COM
I20260806 10:53:17.716514 32666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:17.700883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38714 (local address 127.30.218.2:42477)
0806 10:53:17.701110 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:17.701116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:17.701137 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:53:17.701205 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:17.701208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:17.701327 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:17.701421 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:17.702731 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.703523 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.704883 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.705324 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.705476 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:17.710443 (+ 4967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:17.710463 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:17.710480 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:17.710529 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:17.713358 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:17.714215 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:17.714225 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:17.714228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:17.714388 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:17.714819 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:17.714823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:17.714826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:17.715263 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:17.715639 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:17.715922 (+ 283us) server_negotiation.cc:299] Negotiation successful
0806 10:53:17.716172 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 10:53:17.717422 32509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:17.810463 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 32384
I20260806 10:53:17.827803 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 32520
Aug 06 10:53:17 dist-test-slave-49h3 krb5kdc[32137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.3@KRBTEST.COM
I20260806 10:53:17.845817 32670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:17.828870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50370 (local address 127.30.218.3:35061)
0806 10:53:17.829068 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:17.829073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:17.829098 (+ 25us) server_negotiation.cc:407] Connection header received
0806 10:53:17.829251 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:17.829255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:17.829332 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:17.829474 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:17.830980 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.831814 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.833415 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:17.833962 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:17.834084 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:17.839548 (+ 5464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:17.839566 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:17.839583 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:17.839631 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:17.842774 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:17.843721 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:17.843731 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:17.843733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:17.843905 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:17.844309 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:17.844314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:17.844316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:17.844729 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:17.844984 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:17.845255 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 10:53:17.845505 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 10:53:17.846591 32647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:17.936422 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 32520
I20260806 10:53:17.955236 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 32150
I20260806 10:53:17.956449 32213 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:18.070617 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 32150
2026-08-06T10:53:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4367 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:53:18 dist-test-slave-49h3 krb5kdc[32702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:53:18 dist-test-slave-49h3 krb5kdc[32702](info): set up 2 sockets
Aug 06 10:53:18 dist-test-slave-49h3 krb5kdc[32702](info): commencing operation
krb5kdc: starting...
W20260806 10:53:19.114737 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:53:19 dist-test-slave-49h3 krb5kdc[32702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:53:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:53:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:53:19.342407 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:46855
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:42605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:46855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:19.685583 32715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:19.685941 32715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:19.686107 32715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:19.693802 32715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:53:19.693948 32715 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 10:53:19.694021 32715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:19.694123 32715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:19.694192 32715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:53:19.694289 32715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:53:19.704718 32715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:42605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:46855
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:46855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:19.706079 32715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:19.707412 32715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:19.719080 32721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:19.719143 32724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:19.719121 32722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:19.719724 32715 server_base.cc:1034] running on GCE node
I20260806 10:53:19.720628 32715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:19.722177 32715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:19.723441 32715 hybrid_clock.cc:648] HybridClock initialized: now 1786013599723355 us; error 44 us; skew 500 ppm
I20260806 10:53:19.723815 32715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:19 dist-test-slave-49h3 krb5kdc[32702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013599, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:19.727276 32715 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:53:19.728984 32715 webserver.cc:460] Webserver started at http://127.30.218.62:40733/ using document root <none> and password file <none>
I20260806 10:53:19.729805 32715 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:19.729907 32715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:19.730260 32715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:19.733393 32715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "76a2fe35fdb0415eb3f84bb232c5df0e"
format_stamp: "Formatted at 2026-08-06 10:53:19 on dist-test-slave-49h3"
server_key: "e8119472df7d9767ef945a2da0ef915c"
server_key_iv: "383b4b37616c3868f3939df33eee51df"
server_key_version: "encryptionkey@0"
I20260806 10:53:19.734284 32715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "76a2fe35fdb0415eb3f84bb232c5df0e"
format_stamp: "Formatted at 2026-08-06 10:53:19 on dist-test-slave-49h3"
server_key: "e8119472df7d9767ef945a2da0ef915c"
server_key_iv: "383b4b37616c3868f3939df33eee51df"
server_key_version: "encryptionkey@0"
I20260806 10:53:19.739883 32715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:53:19.743600 32731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:19.744951 32715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:53:19.745127 32715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "76a2fe35fdb0415eb3f84bb232c5df0e"
format_stamp: "Formatted at 2026-08-06 10:53:19 on dist-test-slave-49h3"
server_key: "e8119472df7d9767ef945a2da0ef915c"
server_key_iv: "383b4b37616c3868f3939df33eee51df"
server_key_version: "encryptionkey@0"
I20260806 10:53:19.745342 32715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:19.773638 32715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:19.778059 32715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:19.778362 32715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:19.800680 32715 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:46855
I20260806 10:53:19.800681 327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:46855 every 8 connection(s)
I20260806 10:53:19.802136 32715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:53:19.806087 328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:19.806322 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 32715
I20260806 10:53:19.806536 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:53:19.807191 31592 external_mini_cluster.cc:1428] Setting key c23bbe58f557bd4dc5be70078ac5bb76
I20260806 10:53:19.820408 328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e: Bootstrap starting.
Aug 06 10:53:19 dist-test-slave-49h3 krb5kdc[32702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:19.824370 328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:19.825665 328 log.cc:824] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:19.829139 328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e: No bootstrap required, opened a new log
I20260806 10:53:19.831763 331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:19.809239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52710 (local address 127.30.218.62:46855)
0806 10:53:19.810076 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:19.810087 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:19.810135 (+ 48us) server_negotiation.cc:407] Connection header received
0806 10:53:19.812284 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:19.812309 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:19.812775 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:19.813130 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:19.815001 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:19.816398 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:19.818230 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:19.818944 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:19.819073 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:19.823955 (+ 4882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:19.823963 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:19.823978 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:19.824024 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:19.827545 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:19.828435 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:19.828441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:19.828447 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:19.828624 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:19.829014 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:19.829017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:19.829019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:19.829615 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:19.829850 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:19.830308 (+ 458us) server_negotiation.cc:299] Negotiation successful
0806 10:53:19.830657 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":387,"threads_started":1}
I20260806 10:53:19.837284 328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a2fe35fdb0415eb3f84bb232c5df0e" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46855 } }
I20260806 10:53:19.837769 328 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:19.837880 328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76a2fe35fdb0415eb3f84bb232c5df0e, State: Initialized, Role: FOLLOWER
I20260806 10:53:19.838529 328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a2fe35fdb0415eb3f84bb232c5df0e" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46855 } }
I20260806 10:53:19.838765 328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:53:19.838867 328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:53:19.839004 328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:19.841427 328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a2fe35fdb0415eb3f84bb232c5df0e" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46855 } }
I20260806 10:53:19.842044 328 leader_election.cc:302] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76a2fe35fdb0415eb3f84bb232c5df0e; no voters:
I20260806 10:53:19.842686 328 leader_election.cc:288] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:53:19.842824 333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:19.843542 333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e [term 1 LEADER]: Becoming Leader. State: Replica: 76a2fe35fdb0415eb3f84bb232c5df0e, State: Running, Role: LEADER
I20260806 10:53:19.844125 333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a2fe35fdb0415eb3f84bb232c5df0e" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46855 } }
I20260806 10:53:19.844880 328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:53:19.847021 336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76a2fe35fdb0415eb3f84bb232c5df0e. Latest consensus state: current_term: 1 leader_uuid: "76a2fe35fdb0415eb3f84bb232c5df0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a2fe35fdb0415eb3f84bb232c5df0e" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46855 } } }
I20260806 10:53:19.847476 336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:19.847554 334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76a2fe35fdb0415eb3f84bb232c5df0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a2fe35fdb0415eb3f84bb232c5df0e" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46855 } } }
I20260806 10:53:19.847847 334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:19.849879 339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:53:19.855162 339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:53:19.864859 339 catalog_manager.cc:1347] Generated new cluster ID: 27aed143975e46eb94a3e4eeb462723b
I20260806 10:53:19.865016 339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:53:19.893134 339 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:53:19.894658 339 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:53:19.902566 339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e: Generated new TSK 0
I20260806 10:53:19.903594 339 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
I20260806 10:53:19.970758 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:46855
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:42605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:20.320798 355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:20.321173 355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:20.321317 355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:20.330826 355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:20.330991 355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:20.331122 355 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 10:53:20.331238 355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:53:20.342546 355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:42605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.218.62:46855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:20.343948 355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:20.345436 355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:20.359846 361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:20.359817 360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:20.359791 363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:20.361375 355 server_base.cc:1034] running on GCE node
I20260806 10:53:20.362176 355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:20.363360 355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:20.364624 355 hybrid_clock.cc:648] HybridClock initialized: now 1786013600364548 us; error 45 us; skew 500 ppm
I20260806 10:53:20.364959 355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:20 dist-test-slave-49h3 krb5kdc[32702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013600, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:20.371945 355 init.cc:377] Logged in from keytab as kudu/127.30.218.1@KRBTEST.COM (short username kudu)
I20260806 10:53:20.373795 355 webserver.cc:460] Webserver started at http://127.30.218.1:41691/ using document root <none> and password file <none>
I20260806 10:53:20.374599 355 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:20.374727 355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:20.375072 355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:20.378427 355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "44f59c51d5954d41bd6143fefdbbd813"
format_stamp: "Formatted at 2026-08-06 10:53:20 on dist-test-slave-49h3"
server_key: "32545d06a088cec6177ad6e12fbfa7c2"
server_key_iv: "aa70304a7ea99246305deec3036c4b1f"
server_key_version: "encryptionkey@0"
I20260806 10:53:20.379355 355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "44f59c51d5954d41bd6143fefdbbd813"
format_stamp: "Formatted at 2026-08-06 10:53:20 on dist-test-slave-49h3"
server_key: "32545d06a088cec6177ad6e12fbfa7c2"
server_key_iv: "aa70304a7ea99246305deec3036c4b1f"
server_key_version: "encryptionkey@0"
I20260806 10:53:20.385211 355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 10:53:20.389181 370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:20.390843 355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:53:20.391062 355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "44f59c51d5954d41bd6143fefdbbd813"
format_stamp: "Formatted at 2026-08-06 10:53:20 on dist-test-slave-49h3"
server_key: "32545d06a088cec6177ad6e12fbfa7c2"
server_key_iv: "aa70304a7ea99246305deec3036c4b1f"
server_key_version: "encryptionkey@0"
I20260806 10:53:20.391330 355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:20.418354 355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:20.422791 355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:20.423079 355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:20.426141 355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:20.426256 355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:20.426409 355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:20.426473 355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 10:53:20 dist-test-slave-49h3 krb5kdc[32702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013600, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:20.452714 380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:20.427012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:54269 (local address 127.30.218.62:46855)
0806 10:53:20.427558 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:20.427565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:20.428946 (+ 1381us) server_negotiation.cc:407] Connection header received
0806 10:53:20.432121 (+ 3175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:20.432128 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:20.432236 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:20.432368 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:20.434968 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:20.436213 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:20.438610 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:20.439207 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:20.439363 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:20.445058 (+ 5695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:20.445067 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:20.445072 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:20.445126 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:20.448616 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:20.449923 (+ 1307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:20.449928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:20.449931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:20.450077 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:20.450661 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:20.450664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:20.450666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:20.450959 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:20.451091 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:20.452177 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0806 10:53:20.452383 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":267,"threads_started":1}
I20260806 10:53:20.454916 381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:20.427788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46855 (local address 127.30.218.1:54269)
0806 10:53:20.428724 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:20.428780 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:20.431654 (+ 2874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:20.432718 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:20.432735 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:20.433663 (+ 928us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:20.434699 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:20.434728 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:20.436413 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:20.436418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:20.438407 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:20.438425 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:20.439521 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:20.439530 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:20.439930 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:20.440978 (+ 1048us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:20.441018 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:20.444675 (+ 3657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:20.448814 (+ 4139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:20.448830 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:20.448859 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:20.449763 (+ 904us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:20.450187 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:20.450192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:20.450195 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:20.450549 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:20.451078 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:20.451088 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:20.451704 (+ 616us) client_negotiation.cc:769] Sending connection context
0806 10:53:20.452061 (+ 357us) client_negotiation.cc:240] Negotiation successful
0806 10:53:20.452761 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":341,"threads_started":1}
I20260806 10:53:20.473098 355 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:33557
I20260806 10:53:20.473145 494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:33557 every 8 connection(s)
I20260806 10:53:20.474512 355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:53:20.476946 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 355
I20260806 10:53:20.477185 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:53:20.477057 495 heartbeater.cc:347] Connected to a master server at 127.30.218.62:46855
I20260806 10:53:20.477722 495 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:20.477943 31592 external_mini_cluster.cc:1428] Setting key 187e772c8aa2e4ec3d50fccb05958de8
I20260806 10:53:20.478560 495 heartbeater.cc:511] Master 127.30.218.62:46855 requested a full tablet report, sending...
I20260806 10:53:20.480609 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:20.464924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46855 (local address 127.30.218.1:43511)
0806 10:53:20.465722 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:20.465750 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:20.465953 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:20.466556 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:20.466562 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:20.467291 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:20.467808 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:20.467827 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:20.469276 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:20.469282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:20.470990 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:20.471006 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:20.471829 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:20.471836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:20.471990 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:20.472736 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:20.472756 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:20.473750 (+ 994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:20.477251 (+ 3501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:20.477255 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:20.477258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:20.477894 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:20.478416 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:20.478425 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:20.478429 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:20.478610 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:20.479467 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:20.479471 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:20.479676 (+ 205us) client_negotiation.cc:769] Sending connection context
0806 10:53:20.479810 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 10:53:20.480163 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":381,"threads_started":1}
I20260806 10:53:20.481243 32748 ts_manager.cc:194] Registered new tserver with Master: 44f59c51d5954d41bd6143fefdbbd813 (127.30.218.1:33557)
I20260806 10:53:20.481703 380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:20.464962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:43511 (local address 127.30.218.62:46855)
0806 10:53:20.465203 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:20.465211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:20.465864 (+ 653us) server_negotiation.cc:407] Connection header received
0806 10:53:20.466133 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:20.466141 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:20.466235 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:20.466344 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:20.468057 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:20.469059 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:20.471157 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:20.471616 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:20.471741 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:20.473969 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:20.473974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:20.473977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:20.474015 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:20.477097 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:20.478067 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:20.478074 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:20.478077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:20.478283 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:20.478766 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:20.478771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:20.478774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:20.479212 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:20.479400 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:20.481045 (+ 1645us) server_negotiation.cc:299] Negotiation successful
0806 10:53:20.481315 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 10:53:20.482915 32748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.1@KRBTEST.COM'} at 127.30.218.1:54269
WARNING: no policy specified for kudu/127.30.218.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
I20260806 10:53:20.528611 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:46855
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:42605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:20.895435 499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:20.895892 499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:20.896052 499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:20.905943 499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:20.906147 499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:20.906307 499 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 10:53:20.906435 499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:53:20.920437 499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:42605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.218.62:46855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:20.922041 499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:20.923873 499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:20.941656 507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:20.941674 505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:20.942057 504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:20.942780 499 server_base.cc:1034] running on GCE node
I20260806 10:53:20.943640 499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:20.945048 499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:20.946511 499 hybrid_clock.cc:648] HybridClock initialized: now 1786013600946385 us; error 91 us; skew 500 ppm
I20260806 10:53:20.946956 499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:20 dist-test-slave-49h3 krb5kdc[32702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013600, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:20.951031 499 init.cc:377] Logged in from keytab as kudu/127.30.218.2@KRBTEST.COM (short username kudu)
I20260806 10:53:20.953027 499 webserver.cc:460] Webserver started at http://127.30.218.2:34763/ using document root <none> and password file <none>
I20260806 10:53:20.953964 499 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:20.954103 499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:20.954457 499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:20.958099 499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "06c0ba1c6bca427dba44782f241b00bb"
format_stamp: "Formatted at 2026-08-06 10:53:20 on dist-test-slave-49h3"
server_key: "d3893b3b0765713371925e2f2b3ca1a1"
server_key_iv: "ffb9fe76c8d902dffe0f23010d3658a4"
server_key_version: "encryptionkey@0"
I20260806 10:53:20.959334 499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "06c0ba1c6bca427dba44782f241b00bb"
format_stamp: "Formatted at 2026-08-06 10:53:20 on dist-test-slave-49h3"
server_key: "d3893b3b0765713371925e2f2b3ca1a1"
server_key_iv: "ffb9fe76c8d902dffe0f23010d3658a4"
server_key_version: "encryptionkey@0"
I20260806 10:53:20.966444 499 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260806 10:53:20.970762 514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:20.972862 499 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 10:53:20.973094 499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "06c0ba1c6bca427dba44782f241b00bb"
format_stamp: "Formatted at 2026-08-06 10:53:20 on dist-test-slave-49h3"
server_key: "d3893b3b0765713371925e2f2b3ca1a1"
server_key_iv: "ffb9fe76c8d902dffe0f23010d3658a4"
server_key_version: "encryptionkey@0"
I20260806 10:53:20.973793 499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:20.990567 499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:20.995124 499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:20.995436 499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:20.998425 499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:20.998519 499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:20.998661 499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:20.998742 499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 10:53:21 dist-test-slave-49h3 krb5kdc[32702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013600, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:21.028913 524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:20.999844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46855 (local address 127.30.218.2:39859)
0806 10:53:21.000779 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:21.000829 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:21.003118 (+ 2289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:21.003967 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:21.003976 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:21.004524 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:21.005165 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:21.005185 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.006813 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.006820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:21.009150 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:21.009180 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.010606 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.010616 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:21.010851 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:21.011921 (+ 1070us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:21.011966 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:21.017228 (+ 5262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:21.022422 (+ 5194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:21.022435 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:21.022463 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:21.023472 (+ 1009us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:21.024179 (+ 707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:21.024186 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:21.024199 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:21.024535 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:21.025373 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:21.025383 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:21.026008 (+ 625us) client_negotiation.cc:769] Sending connection context
0806 10:53:21.026424 (+ 416us) client_negotiation.cc:240] Negotiation successful
0806 10:53:21.027141 (+ 717us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":387,"threads_started":1}
I20260806 10:53:21.030607 523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:20.999227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:39859 (local address 127.30.218.62:46855)
0806 10:53:21.001685 (+ 2458us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:21.001691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:21.001719 (+ 28us) server_negotiation.cc:407] Connection header received
0806 10:53:21.003470 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:21.003478 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:21.003576 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:21.003724 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:21.005358 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.006604 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.009370 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.010095 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.010252 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:21.017725 (+ 7473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:21.017733 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:21.017738 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:21.017797 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:21.022115 (+ 4318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:21.023758 (+ 1643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:21.023767 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:21.023770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:21.023985 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:21.024702 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:21.024708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:21.024711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:21.025151 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:21.025369 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:21.029766 (+ 4397us) server_negotiation.cc:299] Negotiation successful
0806 10:53:21.030143 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2308,"thread_start_us":2213,"threads_started":1}
I20260806 10:53:21.048090 499 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:35875
I20260806 10:53:21.048128 637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:35875 every 8 connection(s)
I20260806 10:53:21.050418 499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:53:21.053156 638 heartbeater.cc:347] Connected to a master server at 127.30.218.62:46855
I20260806 10:53:21.053489 638 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:21.054454 638 heartbeater.cc:511] Master 127.30.218.62:46855 requested a full tablet report, sending...
I20260806 10:53:21.054548 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 499
I20260806 10:53:21.054766 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:53:21.055203 31592 external_mini_cluster.cc:1428] Setting key f9a311112d4f5b195bb8740501168b8b
I20260806 10:53:21.055502 611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:21.037294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46855 (local address 127.30.218.2:42673)
0806 10:53:21.038366 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:21.038390 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:21.038592 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:21.039156 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:21.039163 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:21.039895 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:21.040505 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:21.040528 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.042324 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.042331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:21.044449 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:21.044473 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.045910 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.045916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:21.046098 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:21.046793 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:21.046816 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:21.047896 (+ 1080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:21.052121 (+ 4225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:21.052128 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:21.052133 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:21.052714 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:21.053225 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:21.053231 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:21.053234 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:21.053417 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:21.054205 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:21.054211 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:21.054447 (+ 236us) client_negotiation.cc:769] Sending connection context
0806 10:53:21.054623 (+ 176us) client_negotiation.cc:240] Negotiation successful
0806 10:53:21.055024 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"thread_start_us":654,"threads_started":1}
I20260806 10:53:21.056382 32748 ts_manager.cc:194] Registered new tserver with Master: 06c0ba1c6bca427dba44782f241b00bb (127.30.218.2:35875)
I20260806 10:53:21.056561 523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:21.037276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:42673 (local address 127.30.218.62:46855)
0806 10:53:21.037514 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:21.037521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:21.038635 (+ 1114us) server_negotiation.cc:407] Connection header received
0806 10:53:21.038784 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:21.038790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:21.038914 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:21.039059 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:21.040824 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.042098 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.044705 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.045357 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.045621 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:21.048165 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:21.048171 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:21.048175 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:21.048216 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:21.051871 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:21.052849 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:21.052856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:21.052858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:21.053053 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:21.053609 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:21.053616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:21.053619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:21.054048 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:21.054216 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:21.055841 (+ 1625us) server_negotiation.cc:299] Negotiation successful
0806 10:53:21.056157 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 10:53:21.057634 32748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.2@KRBTEST.COM'} at 127.30.218.2:39859
WARNING: no policy specified for kudu/127.30.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
I20260806 10:53:21.108418 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:46855
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:42605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:21.465837 642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:21.466220 642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:21.466414 642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:21.475634 642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:21.475817 642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:21.476006 642 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 10:53:21.476146 642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:53:21.487727 642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:42605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.218.62:46855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:21.488559 495 heartbeater.cc:503] Master 127.30.218.62:46855 was elected leader, sending a full tablet report...
I20260806 10:53:21.489794 642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:21.491678 642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:21.507253 647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:21.507357 648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:21.507287 650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:21.508153 642 server_base.cc:1034] running on GCE node
I20260806 10:53:21.508978 642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:21.510216 642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:21.511485 642 hybrid_clock.cc:648] HybridClock initialized: now 1786013601511393 us; error 55 us; skew 500 ppm
I20260806 10:53:21.511863 642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:21 dist-test-slave-49h3 krb5kdc[32702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013601, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:21.515837 642 init.cc:377] Logged in from keytab as kudu/127.30.218.3@KRBTEST.COM (short username kudu)
I20260806 10:53:21.517685 642 webserver.cc:460] Webserver started at http://127.30.218.3:39159/ using document root <none> and password file <none>
I20260806 10:53:21.518718 642 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:21.518852 642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:21.519259 642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:21.523301 642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "004e7b6702104ebb85e5d3a4dba97cd8"
format_stamp: "Formatted at 2026-08-06 10:53:21 on dist-test-slave-49h3"
server_key: "196ac1bbc89c79f13d2bcebcfc0f6eec"
server_key_iv: "42c95d9c626b405ad228411c518a1591"
server_key_version: "encryptionkey@0"
I20260806 10:53:21.524399 642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "004e7b6702104ebb85e5d3a4dba97cd8"
format_stamp: "Formatted at 2026-08-06 10:53:21 on dist-test-slave-49h3"
server_key: "196ac1bbc89c79f13d2bcebcfc0f6eec"
server_key_iv: "42c95d9c626b405ad228411c518a1591"
server_key_version: "encryptionkey@0"
I20260806 10:53:21.531324 642 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.003s
I20260806 10:53:21.535715 659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:21.537374 642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:53:21.537636 642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "004e7b6702104ebb85e5d3a4dba97cd8"
format_stamp: "Formatted at 2026-08-06 10:53:21 on dist-test-slave-49h3"
server_key: "196ac1bbc89c79f13d2bcebcfc0f6eec"
server_key_iv: "42c95d9c626b405ad228411c518a1591"
server_key_version: "encryptionkey@0"
I20260806 10:53:21.537889 642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:21.557998 642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:21.563643 642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:21.563994 642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:21.567483 642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:21.567592 642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:21.567759 642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:21.567900 642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Aug 06 10:53:21 dist-test-slave-49h3 krb5kdc[32702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013601, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:21.593894 669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:21.568886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46855 (local address 127.30.218.3:44167)
0806 10:53:21.569865 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:21.569925 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:21.572082 (+ 2157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:21.572933 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:21.572942 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:21.573762 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:21.574745 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:21.574777 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.576667 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.576673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:21.578990 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:21.579011 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.579955 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.579962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:21.580226 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:21.581015 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:21.581041 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:21.584232 (+ 3191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:21.588412 (+ 4180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:21.588422 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:21.588441 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:21.589059 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:21.589536 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:21.589542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:21.589545 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:21.589851 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:21.590520 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:21.590528 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:21.591110 (+ 582us) client_negotiation.cc:769] Sending connection context
0806 10:53:21.591440 (+ 330us) client_negotiation.cc:240] Negotiation successful
0806 10:53:21.592080 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":386,"threads_started":1}
I20260806 10:53:21.594667 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:21.568122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:44167 (local address 127.30.218.62:46855)
0806 10:53:21.569006 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:21.569015 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:21.570079 (+ 1064us) server_negotiation.cc:407] Connection header received
0806 10:53:21.572413 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:21.572420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:21.572522 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:21.572635 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:21.575017 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.576461 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.579151 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.579709 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.579839 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:21.584542 (+ 4703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:21.584548 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:21.584552 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:21.584594 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:21.588231 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:21.589191 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:21.589198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:21.589202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:21.589369 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:21.589993 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:21.589996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:21.589999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:21.590397 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:21.590564 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:21.593724 (+ 3160us) server_negotiation.cc:299] Negotiation successful
0806 10:53:21.594076 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":556,"threads_started":1}
I20260806 10:53:21.622399 642 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:45145
I20260806 10:53:21.622478 783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:45145 every 8 connection(s)
I20260806 10:53:21.624452 642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:53:21.624946 737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:21.601259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46855 (local address 127.30.218.3:55597)
0806 10:53:21.609993 (+ 8734us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:21.610026 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:21.610307 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:21.610719 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:21.610724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:21.611200 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:21.611535 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:21.611545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.612615 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.612621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:21.613832 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:21.613847 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.614782 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.614790 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:21.615040 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:21.615975 (+ 935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:21.616005 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:21.617232 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:21.621902 (+ 4670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:21.621913 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:21.621917 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:21.622570 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:21.623075 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:21.623080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:21.623082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:21.623221 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:21.623956 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:21.623959 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:21.624130 (+ 171us) client_negotiation.cc:769] Sending connection context
0806 10:53:21.624246 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 10:53:21.624552 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8555,"thread_start_us":8401,"threads_started":1}
I20260806 10:53:21.625124 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:21.601259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:55597 (local address 127.30.218.62:46855)
0806 10:53:21.601520 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:21.601528 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:21.610142 (+ 8614us) server_negotiation.cc:407] Connection header received
0806 10:53:21.610504 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:21.610510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:21.610586 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:21.610674 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:21.611678 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.612475 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.613978 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.614515 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.614667 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:21.617544 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:21.617553 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:21.617558 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:21.617609 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:21.621726 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:21.622724 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:21.622731 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:21.622734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:21.622935 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:21.623329 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:21.623335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:21.623338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:21.623830 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:21.624024 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:21.624388 (+ 364us) server_negotiation.cc:299] Negotiation successful
0806 10:53:21.624658 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 10:53:21.631078 784 heartbeater.cc:347] Connected to a master server at 127.30.218.62:46855
I20260806 10:53:21.631356 784 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:21.632158 784 heartbeater.cc:511] Master 127.30.218.62:46855 requested a full tablet report, sending...
I20260806 10:53:21.633759 32748 ts_manager.cc:194] Registered new tserver with Master: 004e7b6702104ebb85e5d3a4dba97cd8 (127.30.218.3:45145)
I20260806 10:53:21.634274 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 642
I20260806 10:53:21.634505 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:53:21.634832 32748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.3@KRBTEST.COM'} at 127.30.218.3:44167
I20260806 10:53:21.635051 31592 external_mini_cluster.cc:1428] Setting key 3340eb91e2b653db1701e496d62544c6
I20260806 10:53:21.638367 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 10:53:21 dist-test-slave-49h3 krb5kdc[32702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013601, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 10:53:21 dist-test-slave-49h3 krb5kdc[32702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013601, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:21.700125 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:21.684987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52718 (local address 127.30.218.62:46855)
0806 10:53:21.685213 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:21.685220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:21.685315 (+ 95us) server_negotiation.cc:407] Connection header received
0806 10:53:21.685603 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:21.685609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:21.685691 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:21.685792 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:21.687392 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.688215 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.689609 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.690104 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.690228 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:21.694593 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:21.694598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:21.694601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:21.694641 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:21.697468 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:21.698084 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:21.698088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:21.698091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:21.698245 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:21.698629 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:21.698633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:21.698635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:21.698934 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:21.699231 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:21.699509 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 10:53:21.699763 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 10:53:21.704857 32748 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:53:21.712541 32748 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:53:21.736702 798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:21.726903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:33557 (local address 127.0.0.1:34746)
0806 10:53:21.727800 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:21.727848 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:21.728019 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:21.729093 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:21.729100 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:21.729133 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:53:21.730255 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:21.730271 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.732241 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.732247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:21.734309 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:21.734323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.735651 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.735661 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:21.735886 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:21.735945 (+ 59us) client_negotiation.cc:769] Sending connection context
0806 10:53:21.736080 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 10:53:21.736290 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":445,"threads_started":1}
I20260806 10:53:21.736850 799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:21.727104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34746 (local address 127.30.218.1:33557)
0806 10:53:21.727867 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:21.727876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:21.727962 (+ 86us) server_negotiation.cc:407] Connection header received
0806 10:53:21.728240 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:21.728250 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:21.728563 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:21.728791 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:53:21.730474 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.732065 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.734456 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.735448 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.735584 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:21.736252 (+ 668us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:21.736355 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 10:53:21.736542 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":427,"threads_started":1}
I20260806 10:53:21.738993 800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:21.728247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:35875 (local address 127.0.0.1:55318)
0806 10:53:21.728791 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:21.728811 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:21.728959 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:21.731350 (+ 2391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:21.731355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:21.731380 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:53:21.731842 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:21.731853 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.734006 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.734011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:21.736229 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:21.736242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.738169 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.738176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:21.738341 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:21.738375 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 10:53:21.738439 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 10:53:21.738568 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":21,"thread_start_us":282,"threads_started":1}
I20260806 10:53:21.739482 802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:21.728857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:45145 (local address 127.0.0.1:60974)
0806 10:53:21.729557 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:21.729574 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:21.729680 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:21.731168 (+ 1488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:21.731173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:21.731197 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:53:21.731690 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:21.731699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.733837 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.733842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:21.736470 (+ 2628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:21.736481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.738677 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.738683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:21.738860 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:21.738891 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 10:53:21.738999 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 10:53:21.739118 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":1,"thread_start_us":877,"threads_started":1}
I20260806 10:53:21.739595 803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:21.729533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60974 (local address 127.30.218.3:45145)
0806 10:53:21.730677 (+ 1144us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:21.730683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:21.730712 (+ 29us) server_negotiation.cc:407] Connection header received
0806 10:53:21.730796 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:21.730802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:21.730974 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:21.731145 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:53:21.731895 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.733653 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.736645 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.738176 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.738293 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:21.738537 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:21.739065 (+ 528us) server_negotiation.cc:299] Negotiation successful
0806 10:53:21.739238 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":968,"thread_start_us":733,"threads_started":1}
I20260806 10:53:21.740295 801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:21.728547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55318 (local address 127.30.218.2:35875)
0806 10:53:21.730755 (+ 2208us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:21.730763 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:21.730791 (+ 28us) server_negotiation.cc:407] Connection header received
0806 10:53:21.730905 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:21.730912 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:21.731135 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:21.731914 (+ 779us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:53:21.732179 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.733849 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.736375 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.737917 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.739601 (+ 1684us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:21.739795 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:21.739891 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 10:53:21.740068 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2036,"thread_start_us":1349,"threads_started":1}
I20260806 10:53:21.741901 423 tablet_service.cc:1467] Processing CreateTablet for tablet 9092d8db3a5b4e85908db6e0fb871a97 (DEFAULT_TABLE table=test-table [id=adcc85a63cb049b79c6897e9f9641fba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:21.743350 712 tablet_service.cc:1467] Processing CreateTablet for tablet 9092d8db3a5b4e85908db6e0fb871a97 (DEFAULT_TABLE table=test-table [id=adcc85a63cb049b79c6897e9f9641fba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:21.743901 423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9092d8db3a5b4e85908db6e0fb871a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:21.743774 567 tablet_service.cc:1467] Processing CreateTablet for tablet 9092d8db3a5b4e85908db6e0fb871a97 (DEFAULT_TABLE table=test-table [id=adcc85a63cb049b79c6897e9f9641fba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:21.744685 712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9092d8db3a5b4e85908db6e0fb871a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:21.745137 567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9092d8db3a5b4e85908db6e0fb871a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:21.759763 805 tablet_bootstrap.cc:492] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813: Bootstrap starting.
I20260806 10:53:21.762727 804 tablet_bootstrap.cc:492] T 9092d8db3a5b4e85908db6e0fb871a97 P 004e7b6702104ebb85e5d3a4dba97cd8: Bootstrap starting.
I20260806 10:53:21.764189 805 tablet_bootstrap.cc:654] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:21.765740 805 log.cc:824] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:21.766203 806 tablet_bootstrap.cc:492] T 9092d8db3a5b4e85908db6e0fb871a97 P 06c0ba1c6bca427dba44782f241b00bb: Bootstrap starting.
I20260806 10:53:21.766743 804 tablet_bootstrap.cc:654] T 9092d8db3a5b4e85908db6e0fb871a97 P 004e7b6702104ebb85e5d3a4dba97cd8: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:21.767804 804 log.cc:824] T 9092d8db3a5b4e85908db6e0fb871a97 P 004e7b6702104ebb85e5d3a4dba97cd8: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:21.769697 806 tablet_bootstrap.cc:654] T 9092d8db3a5b4e85908db6e0fb871a97 P 06c0ba1c6bca427dba44782f241b00bb: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:21.770896 805 tablet_bootstrap.cc:492] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813: No bootstrap required, opened a new log
I20260806 10:53:21.770959 806 log.cc:824] T 9092d8db3a5b4e85908db6e0fb871a97 P 06c0ba1c6bca427dba44782f241b00bb: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:21.771070 804 tablet_bootstrap.cc:492] T 9092d8db3a5b4e85908db6e0fb871a97 P 004e7b6702104ebb85e5d3a4dba97cd8: No bootstrap required, opened a new log
I20260806 10:53:21.771184 805 ts_tablet_manager.cc:1397] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260806 10:53:21.771328 804 ts_tablet_manager.cc:1397] T 9092d8db3a5b4e85908db6e0fb871a97 P 004e7b6702104ebb85e5d3a4dba97cd8: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260806 10:53:21.780786 805 raft_consensus.cc:348] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:21.781231 806 tablet_bootstrap.cc:492] T 9092d8db3a5b4e85908db6e0fb871a97 P 06c0ba1c6bca427dba44782f241b00bb: No bootstrap required, opened a new log
I20260806 10:53:21.781302 805 raft_consensus.cc:374] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:21.781481 805 raft_consensus.cc:729] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f59c51d5954d41bd6143fefdbbd813, State: Initialized, Role: FOLLOWER
I20260806 10:53:21.781667 806 ts_tablet_manager.cc:1397] T 9092d8db3a5b4e85908db6e0fb871a97 P 06c0ba1c6bca427dba44782f241b00bb: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.010s
I20260806 10:53:21.782121 804 raft_consensus.cc:348] T 9092d8db3a5b4e85908db6e0fb871a97 P 004e7b6702104ebb85e5d3a4dba97cd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:21.782397 805 consensus_queue.cc:260] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:21.782595 804 raft_consensus.cc:374] T 9092d8db3a5b4e85908db6e0fb871a97 P 004e7b6702104ebb85e5d3a4dba97cd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:21.782701 804 raft_consensus.cc:729] T 9092d8db3a5b4e85908db6e0fb871a97 P 004e7b6702104ebb85e5d3a4dba97cd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 004e7b6702104ebb85e5d3a4dba97cd8, State: Initialized, Role: FOLLOWER
I20260806 10:53:21.783448 804 consensus_queue.cc:260] T 9092d8db3a5b4e85908db6e0fb871a97 P 004e7b6702104ebb85e5d3a4dba97cd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:21.784442 805 ts_tablet_manager.cc:1428] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260806 10:53:21.790227 784 heartbeater.cc:503] Master 127.30.218.62:46855 was elected leader, sending a full tablet report...
I20260806 10:53:21.791015 804 ts_tablet_manager.cc:1428] T 9092d8db3a5b4e85908db6e0fb871a97 P 004e7b6702104ebb85e5d3a4dba97cd8: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
I20260806 10:53:21.793085 806 raft_consensus.cc:348] T 9092d8db3a5b4e85908db6e0fb871a97 P 06c0ba1c6bca427dba44782f241b00bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:21.793474 806 raft_consensus.cc:374] T 9092d8db3a5b4e85908db6e0fb871a97 P 06c0ba1c6bca427dba44782f241b00bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:21.793569 806 raft_consensus.cc:729] T 9092d8db3a5b4e85908db6e0fb871a97 P 06c0ba1c6bca427dba44782f241b00bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c0ba1c6bca427dba44782f241b00bb, State: Initialized, Role: FOLLOWER
I20260806 10:53:21.794256 806 consensus_queue.cc:260] T 9092d8db3a5b4e85908db6e0fb871a97 P 06c0ba1c6bca427dba44782f241b00bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:21.795384 638 heartbeater.cc:503] Master 127.30.218.62:46855 was elected leader, sending a full tablet report...
I20260806 10:53:21.796044 806 ts_tablet_manager.cc:1428] T 9092d8db3a5b4e85908db6e0fb871a97 P 06c0ba1c6bca427dba44782f241b00bb: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
W20260806 10:53:21.802820 639 tablet.cc:2369] T 9092d8db3a5b4e85908db6e0fb871a97 P 06c0ba1c6bca427dba44782f241b00bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:53:21.880913 785 tablet.cc:2369] T 9092d8db3a5b4e85908db6e0fb871a97 P 004e7b6702104ebb85e5d3a4dba97cd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:53:21.924985 810 raft_consensus.cc:482] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:53:21.925406 810 raft_consensus.cc:504] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:21.927570 810 leader_election.cc:288] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06c0ba1c6bca427dba44782f241b00bb (127.30.218.2:35875), 004e7b6702104ebb85e5d3a4dba97cd8 (127.30.218.3:45145)
I20260806 10:53:21.936594 801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:21.928069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:58143 (local address 127.30.218.2:35875)
0806 10:53:21.928283 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:21.928288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:21.928941 (+ 653us) server_negotiation.cc:407] Connection header received
0806 10:53:21.929189 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:21.929195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:21.929265 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:21.929353 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:53:21.930374 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.931485 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.934056 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.935229 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.935439 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:21.935534 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:21.935992 (+ 458us) server_negotiation.cc:299] Negotiation successful
0806 10:53:21.936152 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 10:53:21.936888 803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:21.929382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:48969 (local address 127.30.218.3:45145)
0806 10:53:21.929668 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:21.929676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:21.930200 (+ 524us) server_negotiation.cc:407] Connection header received
0806 10:53:21.930388 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:21.930394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:21.930491 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:21.930611 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:53:21.931072 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.932057 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.934393 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.935672 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.935866 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:21.935947 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:21.936366 (+ 419us) server_negotiation.cc:299] Negotiation successful
0806 10:53:21.936517 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260806 10:53:21.937435 813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:21.928086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:35875 (local address 127.30.218.1:58143)
0806 10:53:21.928800 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:21.928825 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:21.929005 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:21.929383 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:21.929389 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:21.929422 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:53:21.930208 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:21.930223 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.931623 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.931628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:21.933863 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:21.933888 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.935550 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.935558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:21.935770 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:21.935809 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 10:53:21.936873 (+ 1064us) client_negotiation.cc:240] Negotiation successful
0806 10:53:21.937026 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":402,"threads_started":1}
I20260806 10:53:21.937727 814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:21.929003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:45145 (local address 127.30.218.1:48969)
0806 10:53:21.930078 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:21.930095 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:21.930242 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:21.930607 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:21.930610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:21.930629 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:53:21.930926 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:21.930933 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.932205 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.932211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:21.934235 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:21.934245 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.935983 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.935991 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:21.936161 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:21.936189 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 10:53:21.937150 (+ 961us) client_negotiation.cc:240] Negotiation successful
0806 10:53:21.937292 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"thread_start_us":667,"threads_started":1}
I20260806 10:53:21.938859 587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9092d8db3a5b4e85908db6e0fb871a97" candidate_uuid: "44f59c51d5954d41bd6143fefdbbd813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c0ba1c6bca427dba44782f241b00bb" is_pre_election: true
I20260806 10:53:21.939577 587 raft_consensus.cc:2454] T 9092d8db3a5b4e85908db6e0fb871a97 P 06c0ba1c6bca427dba44782f241b00bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f59c51d5954d41bd6143fefdbbd813 in term 0.
I20260806 10:53:21.940176 738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9092d8db3a5b4e85908db6e0fb871a97" candidate_uuid: "44f59c51d5954d41bd6143fefdbbd813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" is_pre_election: true
I20260806 10:53:21.940395 373 leader_election.cc:302] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c0ba1c6bca427dba44782f241b00bb, 44f59c51d5954d41bd6143fefdbbd813; no voters:
I20260806 10:53:21.940855 738 raft_consensus.cc:2454] T 9092d8db3a5b4e85908db6e0fb871a97 P 004e7b6702104ebb85e5d3a4dba97cd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f59c51d5954d41bd6143fefdbbd813 in term 0.
I20260806 10:53:21.940881 810 raft_consensus.cc:2781] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:53:21.941041 810 raft_consensus.cc:482] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:53:21.941141 810 raft_consensus.cc:3037] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:21.944099 810 raft_consensus.cc:504] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:21.944737 810 leader_election.cc:288] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813 [CANDIDATE]: Term 1 election: Requested vote from peers 06c0ba1c6bca427dba44782f241b00bb (127.30.218.2:35875), 004e7b6702104ebb85e5d3a4dba97cd8 (127.30.218.3:45145)
I20260806 10:53:21.945238 587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9092d8db3a5b4e85908db6e0fb871a97" candidate_uuid: "44f59c51d5954d41bd6143fefdbbd813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c0ba1c6bca427dba44782f241b00bb"
I20260806 10:53:21.945465 587 raft_consensus.cc:3037] T 9092d8db3a5b4e85908db6e0fb871a97 P 06c0ba1c6bca427dba44782f241b00bb [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:21.945407 738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9092d8db3a5b4e85908db6e0fb871a97" candidate_uuid: "44f59c51d5954d41bd6143fefdbbd813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "004e7b6702104ebb85e5d3a4dba97cd8"
I20260806 10:53:21.945660 738 raft_consensus.cc:3037] T 9092d8db3a5b4e85908db6e0fb871a97 P 004e7b6702104ebb85e5d3a4dba97cd8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:21.948043 587 raft_consensus.cc:2454] T 9092d8db3a5b4e85908db6e0fb871a97 P 06c0ba1c6bca427dba44782f241b00bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f59c51d5954d41bd6143fefdbbd813 in term 1.
I20260806 10:53:21.948156 738 raft_consensus.cc:2454] T 9092d8db3a5b4e85908db6e0fb871a97 P 004e7b6702104ebb85e5d3a4dba97cd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f59c51d5954d41bd6143fefdbbd813 in term 1.
I20260806 10:53:21.948490 373 leader_election.cc:302] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c0ba1c6bca427dba44782f241b00bb, 44f59c51d5954d41bd6143fefdbbd813; no voters:
I20260806 10:53:21.948755 810 raft_consensus.cc:2781] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:21.949247 810 raft_consensus.cc:686] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813 [term 1 LEADER]: Becoming Leader. State: Replica: 44f59c51d5954d41bd6143fefdbbd813, State: Running, Role: LEADER
I20260806 10:53:21.949823 810 consensus_queue.cc:237] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:21.955176 32748 catalog_manager.cc:5495] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44f59c51d5954d41bd6143fefdbbd813 (127.30.218.1). New cstate: current_term: 1 leader_uuid: "44f59c51d5954d41bd6143fefdbbd813" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } health_report { overall_health: UNKNOWN } } }
W20260806 10:53:21.977434 496 tablet.cc:2369] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:53:21.986296 799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:21.979737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34762 (local address 127.30.218.1:33557)
0806 10:53:21.979935 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:21.979941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:21.980071 (+ 130us) server_negotiation.cc:407] Connection header received
0806 10:53:21.980430 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:21.980435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:21.980568 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:21.980694 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:21.981160 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.982105 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.983601 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:21.984100 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:21.984214 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:21.984832 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:21.985628 (+ 796us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:21.985758 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:21.985879 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 10:53:21.985973 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 10:53:21.996634 738 raft_consensus.cc:1261] T 9092d8db3a5b4e85908db6e0fb871a97 P 004e7b6702104ebb85e5d3a4dba97cd8 [term 1 FOLLOWER]: Refusing update from remote peer 44f59c51d5954d41bd6143fefdbbd813: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:53:21.996619 587 raft_consensus.cc:1261] T 9092d8db3a5b4e85908db6e0fb871a97 P 06c0ba1c6bca427dba44782f241b00bb [term 1 FOLLOWER]: Refusing update from remote peer 44f59c51d5954d41bd6143fefdbbd813: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:53:21.997826 815 consensus_queue.cc:1035] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813 [LEADER]: Connected to new peer: Peer: permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:53:21.998288 810 consensus_queue.cc:1035] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:53:22.011471 820 mvcc.cc:205] Tried to move back new op lower bound from 7315511713766330368 to 7315511713590800384. Current Snapshot: MvccSnapshot[applied={T|T < 7315511713766330368}]
I20260806 10:53:22.011601 818 mvcc.cc:205] Tried to move back new op lower bound from 7315511713766330368 to 7315511713590800384. Current Snapshot: MvccSnapshot[applied={T|T < 7315511713766330368}]
I20260806 10:53:22.033421 32746 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:53:22.033880 32746 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:53:22.038832 32746 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e: Sending DeleteTablet for 3 replicas of tablet 9092d8db3a5b4e85908db6e0fb871a97
I20260806 10:53:22.040287 423 tablet_service.cc:1514] Processing DeleteTablet for tablet 9092d8db3a5b4e85908db6e0fb871a97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:53:22 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:34746
I20260806 10:53:22.040529 567 tablet_service.cc:1514] Processing DeleteTablet for tablet 9092d8db3a5b4e85908db6e0fb871a97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:53:22 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:55318
I20260806 10:53:22.040529 712 tablet_service.cc:1514] Processing DeleteTablet for tablet 9092d8db3a5b4e85908db6e0fb871a97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:53:22 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:60974
I20260806 10:53:22.045714 32746 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:53:22.055248 32746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:53:22.056643 827 tablet_replica.cc:330] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813: stopping tablet replica
I20260806 10:53:22.057667 827 raft_consensus.cc:2229] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:53:22.058804 827 raft_consensus.cc:2258] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:53:22.061708 827 ts_tablet_manager.cc:1905] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:53:22.070026 828 tablet_replica.cc:330] T 9092d8db3a5b4e85908db6e0fb871a97 P 004e7b6702104ebb85e5d3a4dba97cd8: stopping tablet replica
I20260806 10:53:22.070433 423 tablet_service.cc:1467] Processing CreateTablet for tablet 596da221dc99494c9d449bdc66c28fc6 (DEFAULT_TABLE table=test-table [id=2e3555e00de54316a4b4589a3a22e206]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:22.070434 712 tablet_service.cc:1467] Processing CreateTablet for tablet 596da221dc99494c9d449bdc66c28fc6 (DEFAULT_TABLE table=test-table [id=2e3555e00de54316a4b4589a3a22e206]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:22.070688 828 raft_consensus.cc:2229] T 9092d8db3a5b4e85908db6e0fb871a97 P 004e7b6702104ebb85e5d3a4dba97cd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:53:22.070999 712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 596da221dc99494c9d449bdc66c28fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:22.071050 423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 596da221dc99494c9d449bdc66c28fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:22.071441 828 raft_consensus.cc:2258] T 9092d8db3a5b4e85908db6e0fb871a97 P 004e7b6702104ebb85e5d3a4dba97cd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:53:22.072419 829 tablet_replica.cc:330] T 9092d8db3a5b4e85908db6e0fb871a97 P 06c0ba1c6bca427dba44782f241b00bb: stopping tablet replica
I20260806 10:53:22.072868 829 raft_consensus.cc:2229] T 9092d8db3a5b4e85908db6e0fb871a97 P 06c0ba1c6bca427dba44782f241b00bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:53:22.073392 829 raft_consensus.cc:2258] T 9092d8db3a5b4e85908db6e0fb871a97 P 06c0ba1c6bca427dba44782f241b00bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:53:22.074546 566 tablet_service.cc:1467] Processing CreateTablet for tablet 596da221dc99494c9d449bdc66c28fc6 (DEFAULT_TABLE table=test-table [id=2e3555e00de54316a4b4589a3a22e206]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:22.075083 566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 596da221dc99494c9d449bdc66c28fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:22.079216 804 tablet_bootstrap.cc:492] T 596da221dc99494c9d449bdc66c28fc6 P 004e7b6702104ebb85e5d3a4dba97cd8: Bootstrap starting.
I20260806 10:53:22.079216 805 tablet_bootstrap.cc:492] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813: Bootstrap starting.
I20260806 10:53:22.080567 827 ts_tablet_manager.cc:1918] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:53:22.080761 827 log.cc:1196] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-0/wal/wals/9092d8db3a5b4e85908db6e0fb871a97
I20260806 10:53:22.081251 805 tablet_bootstrap.cc:654] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:22.081274 804 tablet_bootstrap.cc:654] T 596da221dc99494c9d449bdc66c28fc6 P 004e7b6702104ebb85e5d3a4dba97cd8: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:22.081421 827 ts_tablet_manager.cc:1939] T 9092d8db3a5b4e85908db6e0fb871a97 P 44f59c51d5954d41bd6143fefdbbd813: Deleting consensus metadata
I20260806 10:53:22.084229 32734 catalog_manager.cc:4841] TS 44f59c51d5954d41bd6143fefdbbd813 (127.30.218.1:33557): tablet 9092d8db3a5b4e85908db6e0fb871a97 (table test-table [id=adcc85a63cb049b79c6897e9f9641fba]) successfully deleted
I20260806 10:53:22.085072 829 ts_tablet_manager.cc:1905] T 9092d8db3a5b4e85908db6e0fb871a97 P 06c0ba1c6bca427dba44782f241b00bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:53:22.087093 805 tablet_bootstrap.cc:492] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813: No bootstrap required, opened a new log
I20260806 10:53:22.087298 805 ts_tablet_manager.cc:1397] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 10:53:22.087369 828 ts_tablet_manager.cc:1905] T 9092d8db3a5b4e85908db6e0fb871a97 P 004e7b6702104ebb85e5d3a4dba97cd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:53:22.088506 805 raft_consensus.cc:348] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:22.088825 806 tablet_bootstrap.cc:492] T 596da221dc99494c9d449bdc66c28fc6 P 06c0ba1c6bca427dba44782f241b00bb: Bootstrap starting.
I20260806 10:53:22.089030 805 raft_consensus.cc:374] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:22.089174 805 raft_consensus.cc:729] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f59c51d5954d41bd6143fefdbbd813, State: Initialized, Role: FOLLOWER
I20260806 10:53:22.089604 805 consensus_queue.cc:260] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:22.091355 806 tablet_bootstrap.cc:654] T 596da221dc99494c9d449bdc66c28fc6 P 06c0ba1c6bca427dba44782f241b00bb: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:22.094264 805 ts_tablet_manager.cc:1428] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20260806 10:53:22.097157 804 tablet_bootstrap.cc:492] T 596da221dc99494c9d449bdc66c28fc6 P 004e7b6702104ebb85e5d3a4dba97cd8: No bootstrap required, opened a new log
I20260806 10:53:22.097390 804 ts_tablet_manager.cc:1397] T 596da221dc99494c9d449bdc66c28fc6 P 004e7b6702104ebb85e5d3a4dba97cd8: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.000s
I20260806 10:53:22.098491 804 raft_consensus.cc:348] T 596da221dc99494c9d449bdc66c28fc6 P 004e7b6702104ebb85e5d3a4dba97cd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:22.098788 804 raft_consensus.cc:374] T 596da221dc99494c9d449bdc66c28fc6 P 004e7b6702104ebb85e5d3a4dba97cd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:22.098904 804 raft_consensus.cc:729] T 596da221dc99494c9d449bdc66c28fc6 P 004e7b6702104ebb85e5d3a4dba97cd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 004e7b6702104ebb85e5d3a4dba97cd8, State: Initialized, Role: FOLLOWER
I20260806 10:53:22.099270 804 consensus_queue.cc:260] T 596da221dc99494c9d449bdc66c28fc6 P 004e7b6702104ebb85e5d3a4dba97cd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:22.102820 804 ts_tablet_manager.cc:1428] T 596da221dc99494c9d449bdc66c28fc6 P 004e7b6702104ebb85e5d3a4dba97cd8: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260806 10:53:22.103526 829 ts_tablet_manager.cc:1918] T 9092d8db3a5b4e85908db6e0fb871a97 P 06c0ba1c6bca427dba44782f241b00bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:53:22.103793 829 log.cc:1196] T 9092d8db3a5b4e85908db6e0fb871a97 P 06c0ba1c6bca427dba44782f241b00bb: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/9092d8db3a5b4e85908db6e0fb871a97
I20260806 10:53:22.104470 829 ts_tablet_manager.cc:1939] T 9092d8db3a5b4e85908db6e0fb871a97 P 06c0ba1c6bca427dba44782f241b00bb: Deleting consensus metadata
I20260806 10:53:22.106841 32734 catalog_manager.cc:4841] TS 06c0ba1c6bca427dba44782f241b00bb (127.30.218.2:35875): tablet 9092d8db3a5b4e85908db6e0fb871a97 (table test-table [id=adcc85a63cb049b79c6897e9f9641fba]) successfully deleted
I20260806 10:53:22.107796 828 ts_tablet_manager.cc:1918] T 9092d8db3a5b4e85908db6e0fb871a97 P 004e7b6702104ebb85e5d3a4dba97cd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:53:22.107997 828 log.cc:1196] T 9092d8db3a5b4e85908db6e0fb871a97 P 004e7b6702104ebb85e5d3a4dba97cd8: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/9092d8db3a5b4e85908db6e0fb871a97
I20260806 10:53:22.108608 828 ts_tablet_manager.cc:1939] T 9092d8db3a5b4e85908db6e0fb871a97 P 004e7b6702104ebb85e5d3a4dba97cd8: Deleting consensus metadata
I20260806 10:53:22.108636 806 tablet_bootstrap.cc:492] T 596da221dc99494c9d449bdc66c28fc6 P 06c0ba1c6bca427dba44782f241b00bb: No bootstrap required, opened a new log
I20260806 10:53:22.108798 806 ts_tablet_manager.cc:1397] T 596da221dc99494c9d449bdc66c28fc6 P 06c0ba1c6bca427dba44782f241b00bb: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.004s
I20260806 10:53:22.109598 806 raft_consensus.cc:348] T 596da221dc99494c9d449bdc66c28fc6 P 06c0ba1c6bca427dba44782f241b00bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:22.109807 806 raft_consensus.cc:374] T 596da221dc99494c9d449bdc66c28fc6 P 06c0ba1c6bca427dba44782f241b00bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:22.109917 806 raft_consensus.cc:729] T 596da221dc99494c9d449bdc66c28fc6 P 06c0ba1c6bca427dba44782f241b00bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c0ba1c6bca427dba44782f241b00bb, State: Initialized, Role: FOLLOWER
I20260806 10:53:22.110225 806 consensus_queue.cc:260] T 596da221dc99494c9d449bdc66c28fc6 P 06c0ba1c6bca427dba44782f241b00bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:22.111014 806 ts_tablet_manager.cc:1428] T 596da221dc99494c9d449bdc66c28fc6 P 06c0ba1c6bca427dba44782f241b00bb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 10:53:22.111691 32734 catalog_manager.cc:4841] TS 004e7b6702104ebb85e5d3a4dba97cd8 (127.30.218.3:45145): tablet 9092d8db3a5b4e85908db6e0fb871a97 (table test-table [id=adcc85a63cb049b79c6897e9f9641fba]) successfully deleted
I20260806 10:53:22.180846 815 raft_consensus.cc:482] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:53:22.181159 815 raft_consensus.cc:504] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:22.182024 815 leader_election.cc:288] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06c0ba1c6bca427dba44782f241b00bb (127.30.218.2:35875), 004e7b6702104ebb85e5d3a4dba97cd8 (127.30.218.3:45145)
I20260806 10:53:22.182845 738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596da221dc99494c9d449bdc66c28fc6" candidate_uuid: "44f59c51d5954d41bd6143fefdbbd813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" is_pre_election: true
I20260806 10:53:22.182857 587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596da221dc99494c9d449bdc66c28fc6" candidate_uuid: "44f59c51d5954d41bd6143fefdbbd813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c0ba1c6bca427dba44782f241b00bb" is_pre_election: true
I20260806 10:53:22.183145 738 raft_consensus.cc:2454] T 596da221dc99494c9d449bdc66c28fc6 P 004e7b6702104ebb85e5d3a4dba97cd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f59c51d5954d41bd6143fefdbbd813 in term 0.
I20260806 10:53:22.183177 587 raft_consensus.cc:2454] T 596da221dc99494c9d449bdc66c28fc6 P 06c0ba1c6bca427dba44782f241b00bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f59c51d5954d41bd6143fefdbbd813 in term 0.
I20260806 10:53:22.183650 373 leader_election.cc:302] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 004e7b6702104ebb85e5d3a4dba97cd8, 44f59c51d5954d41bd6143fefdbbd813; no voters:
I20260806 10:53:22.184010 815 raft_consensus.cc:2781] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:53:22.184139 815 raft_consensus.cc:482] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:53:22.184262 815 raft_consensus.cc:3037] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:22.186913 815 raft_consensus.cc:504] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:22.187484 815 leader_election.cc:288] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813 [CANDIDATE]: Term 1 election: Requested vote from peers 06c0ba1c6bca427dba44782f241b00bb (127.30.218.2:35875), 004e7b6702104ebb85e5d3a4dba97cd8 (127.30.218.3:45145)
I20260806 10:53:22.188097 738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596da221dc99494c9d449bdc66c28fc6" candidate_uuid: "44f59c51d5954d41bd6143fefdbbd813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "004e7b6702104ebb85e5d3a4dba97cd8"
I20260806 10:53:22.188105 587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596da221dc99494c9d449bdc66c28fc6" candidate_uuid: "44f59c51d5954d41bd6143fefdbbd813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c0ba1c6bca427dba44782f241b00bb"
I20260806 10:53:22.188318 738 raft_consensus.cc:3037] T 596da221dc99494c9d449bdc66c28fc6 P 004e7b6702104ebb85e5d3a4dba97cd8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:22.188323 587 raft_consensus.cc:3037] T 596da221dc99494c9d449bdc66c28fc6 P 06c0ba1c6bca427dba44782f241b00bb [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:22.190582 738 raft_consensus.cc:2454] T 596da221dc99494c9d449bdc66c28fc6 P 004e7b6702104ebb85e5d3a4dba97cd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f59c51d5954d41bd6143fefdbbd813 in term 1.
I20260806 10:53:22.190994 373 leader_election.cc:302] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 004e7b6702104ebb85e5d3a4dba97cd8, 44f59c51d5954d41bd6143fefdbbd813; no voters:
I20260806 10:53:22.191254 815 raft_consensus.cc:2781] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:22.191634 587 raft_consensus.cc:2454] T 596da221dc99494c9d449bdc66c28fc6 P 06c0ba1c6bca427dba44782f241b00bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f59c51d5954d41bd6143fefdbbd813 in term 1.
I20260806 10:53:22.191450 815 raft_consensus.cc:686] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813 [term 1 LEADER]: Becoming Leader. State: Replica: 44f59c51d5954d41bd6143fefdbbd813, State: Running, Role: LEADER
I20260806 10:53:22.192054 815 consensus_queue.cc:237] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:22.202775 32748 catalog_manager.cc:5495] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44f59c51d5954d41bd6143fefdbbd813 (127.30.218.1). New cstate: current_term: 1 leader_uuid: "44f59c51d5954d41bd6143fefdbbd813" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } health_report { overall_health: UNKNOWN } } }
Aug 06 10:53:22 dist-test-slave-49h3 krb5kdc[32702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013599, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:22.265246 840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:22.248549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52730 (local address 127.30.218.62:46855)
0806 10:53:22.249920 (+ 1371us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:22.249925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:22.249943 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:53:22.250001 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:22.250005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:22.250078 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:22.250178 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:22.251805 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:22.252675 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:22.254148 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:22.254604 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:22.254715 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:22.259030 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:22.259036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:22.259039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:22.259074 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:22.262055 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:22.263003 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:22.263007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:22.263009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:22.263137 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:22.263552 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:22.263555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:22.263557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:22.263867 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:22.263984 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:22.264548 (+ 564us) server_negotiation.cc:299] Negotiation successful
0806 10:53:22.264905 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":164,"server-negotiator.queue_time_us":1238,"spinlock_wait_cycles":31104,"thread_start_us":1121,"threads_started":1}
I20260806 10:53:22.265275 839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:22.248537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46855 (local address 127.0.0.1:52730)
0806 10:53:22.249264 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:22.249290 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:22.249588 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:22.250259 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:22.250264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:22.251274 (+ 1010us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:22.251616 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:22.251630 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:22.252808 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:22.252822 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:22.254015 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:22.254030 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:22.254770 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:22.254776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:22.254955 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:22.255795 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:22.255831 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:22.258790 (+ 2959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:22.262206 (+ 3416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:22.262223 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:22.262233 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:22.262870 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:22.263231 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:22.263236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:22.263240 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:22.263447 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:22.263995 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:22.264005 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:22.264241 (+ 236us) client_negotiation.cc:769] Sending connection context
0806 10:53:22.264448 (+ 207us) client_negotiation.cc:240] Negotiation successful
0806 10:53:22.264800 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":336,"threads_started":1}
I20260806 10:53:22.272523 32748 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:52730:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 10:53:22.274470 32748 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:53:22.283591 423 tablet_service.cc:1467] Processing CreateTablet for tablet 638ac97373a84d37b002def41d4a9fcd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7db478f78696496b84e7cf8c5633f647]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 10:53:22.284025 712 tablet_service.cc:1467] Processing CreateTablet for tablet 638ac97373a84d37b002def41d4a9fcd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7db478f78696496b84e7cf8c5633f647]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 10:53:22.284361 423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 638ac97373a84d37b002def41d4a9fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:22.284691 712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 638ac97373a84d37b002def41d4a9fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:22.291177 805 tablet_bootstrap.cc:492] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813: Bootstrap starting.
I20260806 10:53:22.292043 566 tablet_service.cc:1467] Processing CreateTablet for tablet 638ac97373a84d37b002def41d4a9fcd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7db478f78696496b84e7cf8c5633f647]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 10:53:22.292721 566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 638ac97373a84d37b002def41d4a9fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:22.293514 805 tablet_bootstrap.cc:654] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:22.293916 804 tablet_bootstrap.cc:492] T 638ac97373a84d37b002def41d4a9fcd P 004e7b6702104ebb85e5d3a4dba97cd8: Bootstrap starting.
I20260806 10:53:22.295984 804 tablet_bootstrap.cc:654] T 638ac97373a84d37b002def41d4a9fcd P 004e7b6702104ebb85e5d3a4dba97cd8: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:22.302439 806 tablet_bootstrap.cc:492] T 638ac97373a84d37b002def41d4a9fcd P 06c0ba1c6bca427dba44782f241b00bb: Bootstrap starting.
I20260806 10:53:22.303599 804 tablet_bootstrap.cc:492] T 638ac97373a84d37b002def41d4a9fcd P 004e7b6702104ebb85e5d3a4dba97cd8: No bootstrap required, opened a new log
I20260806 10:53:22.303779 804 ts_tablet_manager.cc:1397] T 638ac97373a84d37b002def41d4a9fcd P 004e7b6702104ebb85e5d3a4dba97cd8: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.001s
I20260806 10:53:22.304333 806 tablet_bootstrap.cc:654] T 638ac97373a84d37b002def41d4a9fcd P 06c0ba1c6bca427dba44782f241b00bb: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:22.304536 804 raft_consensus.cc:348] T 638ac97373a84d37b002def41d4a9fcd P 004e7b6702104ebb85e5d3a4dba97cd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:22.304775 804 raft_consensus.cc:374] T 638ac97373a84d37b002def41d4a9fcd P 004e7b6702104ebb85e5d3a4dba97cd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:22.304857 804 raft_consensus.cc:729] T 638ac97373a84d37b002def41d4a9fcd P 004e7b6702104ebb85e5d3a4dba97cd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 004e7b6702104ebb85e5d3a4dba97cd8, State: Initialized, Role: FOLLOWER
I20260806 10:53:22.305130 804 consensus_queue.cc:260] T 638ac97373a84d37b002def41d4a9fcd P 004e7b6702104ebb85e5d3a4dba97cd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:22.305789 804 ts_tablet_manager.cc:1428] T 638ac97373a84d37b002def41d4a9fcd P 004e7b6702104ebb85e5d3a4dba97cd8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 10:53:22.305771 811 tablet_replica.cc:439] T 638ac97373a84d37b002def41d4a9fcd P 004e7b6702104ebb85e5d3a4dba97cd8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } } }
I20260806 10:53:22.306190 811 tablet_replica.cc:442] T 638ac97373a84d37b002def41d4a9fcd P 004e7b6702104ebb85e5d3a4dba97cd8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:53:22.307106 805 tablet_bootstrap.cc:492] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813: No bootstrap required, opened a new log
I20260806 10:53:22.307319 805 ts_tablet_manager.cc:1397] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.010s
I20260806 10:53:22.307894 806 tablet_bootstrap.cc:492] T 638ac97373a84d37b002def41d4a9fcd P 06c0ba1c6bca427dba44782f241b00bb: No bootstrap required, opened a new log
I20260806 10:53:22.308081 806 ts_tablet_manager.cc:1397] T 638ac97373a84d37b002def41d4a9fcd P 06c0ba1c6bca427dba44782f241b00bb: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260806 10:53:22.308235 805 raft_consensus.cc:348] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:22.308554 805 raft_consensus.cc:374] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:22.308673 805 raft_consensus.cc:729] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f59c51d5954d41bd6143fefdbbd813, State: Initialized, Role: FOLLOWER
I20260806 10:53:22.308943 806 raft_consensus.cc:348] T 638ac97373a84d37b002def41d4a9fcd P 06c0ba1c6bca427dba44782f241b00bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:22.308971 805 consensus_queue.cc:260] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:22.309204 806 raft_consensus.cc:374] T 638ac97373a84d37b002def41d4a9fcd P 06c0ba1c6bca427dba44782f241b00bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:22.309293 806 raft_consensus.cc:729] T 638ac97373a84d37b002def41d4a9fcd P 06c0ba1c6bca427dba44782f241b00bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c0ba1c6bca427dba44782f241b00bb, State: Initialized, Role: FOLLOWER
I20260806 10:53:22.309562 815 tablet_replica.cc:439] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } } }
I20260806 10:53:22.309626 806 consensus_queue.cc:260] T 638ac97373a84d37b002def41d4a9fcd P 06c0ba1c6bca427dba44782f241b00bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:22.309890 815 tablet_replica.cc:442] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:53:22.310693 812 tablet_replica.cc:439] T 638ac97373a84d37b002def41d4a9fcd P 06c0ba1c6bca427dba44782f241b00bb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } } }
I20260806 10:53:22.310997 805 ts_tablet_manager.cc:1428] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260806 10:53:22.311079 812 tablet_replica.cc:442] T 638ac97373a84d37b002def41d4a9fcd P 06c0ba1c6bca427dba44782f241b00bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:53:22.312089 806 ts_tablet_manager.cc:1428] T 638ac97373a84d37b002def41d4a9fcd P 06c0ba1c6bca427dba44782f241b00bb: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260806 10:53:22.330505 815 raft_consensus.cc:482] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:53:22.330734 815 raft_consensus.cc:504] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:22.331449 815 leader_election.cc:288] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06c0ba1c6bca427dba44782f241b00bb (127.30.218.2:35875), 004e7b6702104ebb85e5d3a4dba97cd8 (127.30.218.3:45145)
I20260806 10:53:22.332139 587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "638ac97373a84d37b002def41d4a9fcd" candidate_uuid: "44f59c51d5954d41bd6143fefdbbd813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c0ba1c6bca427dba44782f241b00bb" is_pre_election: true
I20260806 10:53:22.332209 738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "638ac97373a84d37b002def41d4a9fcd" candidate_uuid: "44f59c51d5954d41bd6143fefdbbd813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" is_pre_election: true
I20260806 10:53:22.332504 587 raft_consensus.cc:2454] T 638ac97373a84d37b002def41d4a9fcd P 06c0ba1c6bca427dba44782f241b00bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f59c51d5954d41bd6143fefdbbd813 in term 0.
I20260806 10:53:22.332566 738 raft_consensus.cc:2454] T 638ac97373a84d37b002def41d4a9fcd P 004e7b6702104ebb85e5d3a4dba97cd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f59c51d5954d41bd6143fefdbbd813 in term 0.
I20260806 10:53:22.333045 373 leader_election.cc:302] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 004e7b6702104ebb85e5d3a4dba97cd8, 44f59c51d5954d41bd6143fefdbbd813; no voters:
I20260806 10:53:22.333325 815 raft_consensus.cc:2781] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:53:22.333500 815 raft_consensus.cc:482] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:53:22.333657 815 raft_consensus.cc:3037] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:22.336215 815 raft_consensus.cc:504] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:22.336874 815 leader_election.cc:288] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813 [CANDIDATE]: Term 1 election: Requested vote from peers 06c0ba1c6bca427dba44782f241b00bb (127.30.218.2:35875), 004e7b6702104ebb85e5d3a4dba97cd8 (127.30.218.3:45145)
I20260806 10:53:22.337469 587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "638ac97373a84d37b002def41d4a9fcd" candidate_uuid: "44f59c51d5954d41bd6143fefdbbd813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c0ba1c6bca427dba44782f241b00bb"
I20260806 10:53:22.337585 738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "638ac97373a84d37b002def41d4a9fcd" candidate_uuid: "44f59c51d5954d41bd6143fefdbbd813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "004e7b6702104ebb85e5d3a4dba97cd8"
I20260806 10:53:22.337775 587 raft_consensus.cc:3037] T 638ac97373a84d37b002def41d4a9fcd P 06c0ba1c6bca427dba44782f241b00bb [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:22.337883 738 raft_consensus.cc:3037] T 638ac97373a84d37b002def41d4a9fcd P 004e7b6702104ebb85e5d3a4dba97cd8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:22.340082 587 raft_consensus.cc:2454] T 638ac97373a84d37b002def41d4a9fcd P 06c0ba1c6bca427dba44782f241b00bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f59c51d5954d41bd6143fefdbbd813 in term 1.
I20260806 10:53:22.340582 373 leader_election.cc:302] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c0ba1c6bca427dba44782f241b00bb, 44f59c51d5954d41bd6143fefdbbd813; no voters:
I20260806 10:53:22.341006 815 raft_consensus.cc:2781] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:22.341228 815 raft_consensus.cc:686] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813 [term 1 LEADER]: Becoming Leader. State: Replica: 44f59c51d5954d41bd6143fefdbbd813, State: Running, Role: LEADER
I20260806 10:53:22.341295 738 raft_consensus.cc:2454] T 638ac97373a84d37b002def41d4a9fcd P 004e7b6702104ebb85e5d3a4dba97cd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f59c51d5954d41bd6143fefdbbd813 in term 1.
I20260806 10:53:22.341751 815 consensus_queue.cc:237] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } }
I20260806 10:53:22.344074 810 tablet_replica.cc:439] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813: TxnStatusTablet state changed. Reason: New leader 44f59c51d5954d41bd6143fefdbbd813. Latest consensus state: current_term: 1 leader_uuid: "44f59c51d5954d41bd6143fefdbbd813" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } } }
I20260806 10:53:22.344436 810 tablet_replica.cc:442] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813: This TxnStatusTablet replica's current role is: LEADER
I20260806 10:53:22.345388 845 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 10:53:22.345784 845 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 10:53:22.345870 32746 catalog_manager.cc:5495] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44f59c51d5954d41bd6143fefdbbd813 (127.30.218.1). New cstate: current_term: 1 leader_uuid: "44f59c51d5954d41bd6143fefdbbd813" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } health_report { overall_health: UNKNOWN } } }
I20260806 10:53:22.368845 839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:22.362812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:33557 (local address 127.0.0.1:34768)
0806 10:53:22.363077 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:22.363100 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:22.363244 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:22.363679 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:22.363683 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:22.363693 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 10:53:22.363896 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:22.363904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:22.365535 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:22.365542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:22.366639 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:22.366652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:22.367610 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:22.367614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:22.367792 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:22.367865 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 10:53:22.368331 (+ 466us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 10:53:22.368343 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 10:53:22.368402 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 10:53:22.368538 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 10:53:22.368906 799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:22.362840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34768 (local address 127.30.218.1:33557)
0806 10:53:22.363057 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:22.363065 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:22.363176 (+ 111us) server_negotiation.cc:407] Connection header received
0806 10:53:22.363431 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:22.363438 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:22.363544 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:22.363674 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:22.364031 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:22.365358 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:22.366796 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:22.367467 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:22.367623 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:22.368050 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:22.368240 (+ 190us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:22.368341 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:22.368434 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 10:53:22.368534 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 10:53:22.373754 587 raft_consensus.cc:1261] T 638ac97373a84d37b002def41d4a9fcd P 06c0ba1c6bca427dba44782f241b00bb [term 1 FOLLOWER]: Refusing update from remote peer 44f59c51d5954d41bd6143fefdbbd813: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:53:22.373888 738 raft_consensus.cc:1261] T 638ac97373a84d37b002def41d4a9fcd P 004e7b6702104ebb85e5d3a4dba97cd8 [term 1 FOLLOWER]: Refusing update from remote peer 44f59c51d5954d41bd6143fefdbbd813: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:53:22.374538 815 consensus_queue.cc:1035] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813 [LEADER]: Connected to new peer: Peer: permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:53:22.374940 810 consensus_queue.cc:1035] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:53:22.378568 811 tablet_replica.cc:439] T 638ac97373a84d37b002def41d4a9fcd P 004e7b6702104ebb85e5d3a4dba97cd8: TxnStatusTablet state changed. Reason: New leader 44f59c51d5954d41bd6143fefdbbd813. Latest consensus state: current_term: 1 leader_uuid: "44f59c51d5954d41bd6143fefdbbd813" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } } }
I20260806 10:53:22.378937 811 tablet_replica.cc:442] T 638ac97373a84d37b002def41d4a9fcd P 004e7b6702104ebb85e5d3a4dba97cd8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:53:22.382716 810 tablet_replica.cc:439] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44f59c51d5954d41bd6143fefdbbd813" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } } }
I20260806 10:53:22.383060 810 tablet_replica.cc:442] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813: This TxnStatusTablet replica's current role is: LEADER
I20260806 10:53:22.383569 812 tablet_replica.cc:439] T 638ac97373a84d37b002def41d4a9fcd P 06c0ba1c6bca427dba44782f241b00bb: TxnStatusTablet state changed. Reason: New leader 44f59c51d5954d41bd6143fefdbbd813. Latest consensus state: current_term: 1 leader_uuid: "44f59c51d5954d41bd6143fefdbbd813" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } } }
I20260806 10:53:22.383935 812 tablet_replica.cc:442] T 638ac97373a84d37b002def41d4a9fcd P 06c0ba1c6bca427dba44782f241b00bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:53:22.389161 811 tablet_replica.cc:439] T 638ac97373a84d37b002def41d4a9fcd P 004e7b6702104ebb85e5d3a4dba97cd8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44f59c51d5954d41bd6143fefdbbd813" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } } }
I20260806 10:53:22.389509 811 tablet_replica.cc:442] T 638ac97373a84d37b002def41d4a9fcd P 004e7b6702104ebb85e5d3a4dba97cd8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:53:22.390628 812 tablet_replica.cc:439] T 638ac97373a84d37b002def41d4a9fcd P 06c0ba1c6bca427dba44782f241b00bb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44f59c51d5954d41bd6143fefdbbd813" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } } }
I20260806 10:53:22.390899 812 tablet_replica.cc:442] T 638ac97373a84d37b002def41d4a9fcd P 06c0ba1c6bca427dba44782f241b00bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:53:22.387727 849 tablet_replica.cc:439] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44f59c51d5954d41bd6143fefdbbd813" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f59c51d5954d41bd6143fefdbbd813" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33557 } } peers { permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 } } peers { permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 } } }
I20260806 10:53:22.392051 849 tablet_replica.cc:442] T 638ac97373a84d37b002def41d4a9fcd P 44f59c51d5954d41bd6143fefdbbd813: This TxnStatusTablet replica's current role is: LEADER
I20260806 10:53:22.415506 799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:22.408375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:50859 (local address 127.30.218.1:33557)
0806 10:53:22.408624 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:22.408631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:22.409292 (+ 661us) server_negotiation.cc:407] Connection header received
0806 10:53:22.409505 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:22.409512 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:22.409605 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:22.409703 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:22.410363 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:22.411685 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:22.413246 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:22.414008 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:22.414196 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:22.414662 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:22.414824 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:22.414929 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:22.415039 (+ 110us) server_negotiation.cc:299] Negotiation successful
0806 10:53:22.415125 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":89}
I20260806 10:53:22.415558 852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:22.408343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:33557 (local address 127.30.218.1:50859)
0806 10:53:22.409168 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:22.409191 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:22.409353 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:22.409711 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:22.409714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:22.409723 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 10:53:22.410181 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:22.410191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:22.411811 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:22.411820 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:22.413089 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:22.413101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:22.414235 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:22.414241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:22.414399 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:22.414459 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 10:53:22.414924 (+ 465us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 10:53:22.414933 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 10:53:22.414996 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 10:53:22.415108 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":26,"thread_start_us":482,"threads_started":1}
I20260806 10:53:22.429339 738 raft_consensus.cc:1261] T 596da221dc99494c9d449bdc66c28fc6 P 004e7b6702104ebb85e5d3a4dba97cd8 [term 1 FOLLOWER]: Refusing update from remote peer 44f59c51d5954d41bd6143fefdbbd813: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:53:22.429353 587 raft_consensus.cc:1261] T 596da221dc99494c9d449bdc66c28fc6 P 06c0ba1c6bca427dba44782f241b00bb [term 1 FOLLOWER]: Refusing update from remote peer 44f59c51d5954d41bd6143fefdbbd813: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:53:22.430104 810 consensus_queue.cc:1035] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c0ba1c6bca427dba44782f241b00bb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:53:22.430568 815 consensus_queue.cc:1035] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813 [LEADER]: Connected to new peer: Peer: permanent_uuid: "004e7b6702104ebb85e5d3a4dba97cd8" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:53:22.449373 819 mvcc.cc:205] Tried to move back new op lower bound from 7315511715543388160 to 7315511714581618688. Current Snapshot: MvccSnapshot[applied={T|T < 7315511715543388160}]
I20260806 10:53:22.516090 32746 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:53:22.516392 32746 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:53:22.520038 32746 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76a2fe35fdb0415eb3f84bb232c5df0e: Sending DeleteTablet for 3 replicas of tablet 596da221dc99494c9d449bdc66c28fc6
I20260806 10:53:22.521407 422 tablet_service.cc:1514] Processing DeleteTablet for tablet 596da221dc99494c9d449bdc66c28fc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:53:22 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:34746
I20260806 10:53:22.522064 566 tablet_service.cc:1514] Processing DeleteTablet for tablet 596da221dc99494c9d449bdc66c28fc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:53:22 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:55318
I20260806 10:53:22.522040 712 tablet_service.cc:1514] Processing DeleteTablet for tablet 596da221dc99494c9d449bdc66c28fc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:53:22 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:60974
I20260806 10:53:22.522382 829 tablet_replica.cc:330] T 596da221dc99494c9d449bdc66c28fc6 P 06c0ba1c6bca427dba44782f241b00bb: stopping tablet replica
I20260806 10:53:22.522405 827 tablet_replica.cc:330] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813: stopping tablet replica
I20260806 10:53:22.522612 827 raft_consensus.cc:2229] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:53:22.522608 829 raft_consensus.cc:2229] T 596da221dc99494c9d449bdc66c28fc6 P 06c0ba1c6bca427dba44782f241b00bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:53:22.522830 829 raft_consensus.cc:2258] T 596da221dc99494c9d449bdc66c28fc6 P 06c0ba1c6bca427dba44782f241b00bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:53:22.522979 827 raft_consensus.cc:2258] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:53:22.523145 828 tablet_replica.cc:330] T 596da221dc99494c9d449bdc66c28fc6 P 004e7b6702104ebb85e5d3a4dba97cd8: stopping tablet replica
I20260806 10:53:22.523376 828 raft_consensus.cc:2229] T 596da221dc99494c9d449bdc66c28fc6 P 004e7b6702104ebb85e5d3a4dba97cd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:53:22.523602 828 raft_consensus.cc:2258] T 596da221dc99494c9d449bdc66c28fc6 P 004e7b6702104ebb85e5d3a4dba97cd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:53:22.525300 828 ts_tablet_manager.cc:1905] T 596da221dc99494c9d449bdc66c28fc6 P 004e7b6702104ebb85e5d3a4dba97cd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:53:22.525396 827 ts_tablet_manager.cc:1905] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:53:22.530587 829 ts_tablet_manager.cc:1905] T 596da221dc99494c9d449bdc66c28fc6 P 06c0ba1c6bca427dba44782f241b00bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:53:22.535737 828 ts_tablet_manager.cc:1918] T 596da221dc99494c9d449bdc66c28fc6 P 004e7b6702104ebb85e5d3a4dba97cd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:53:22.536239 828 log.cc:1196] T 596da221dc99494c9d449bdc66c28fc6 P 004e7b6702104ebb85e5d3a4dba97cd8: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/596da221dc99494c9d449bdc66c28fc6
I20260806 10:53:22.537014 828 ts_tablet_manager.cc:1939] T 596da221dc99494c9d449bdc66c28fc6 P 004e7b6702104ebb85e5d3a4dba97cd8: Deleting consensus metadata
I20260806 10:53:22.538806 32734 catalog_manager.cc:4841] TS 004e7b6702104ebb85e5d3a4dba97cd8 (127.30.218.3:45145): tablet 596da221dc99494c9d449bdc66c28fc6 (table test-table [id=2e3555e00de54316a4b4589a3a22e206]) successfully deleted
I20260806 10:53:22.543128 829 ts_tablet_manager.cc:1918] T 596da221dc99494c9d449bdc66c28fc6 P 06c0ba1c6bca427dba44782f241b00bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:53:22.543351 829 log.cc:1196] T 596da221dc99494c9d449bdc66c28fc6 P 06c0ba1c6bca427dba44782f241b00bb: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/596da221dc99494c9d449bdc66c28fc6
I20260806 10:53:22.543781 829 ts_tablet_manager.cc:1939] T 596da221dc99494c9d449bdc66c28fc6 P 06c0ba1c6bca427dba44782f241b00bb: Deleting consensus metadata
I20260806 10:53:22.545039 32734 catalog_manager.cc:4841] TS 06c0ba1c6bca427dba44782f241b00bb (127.30.218.2:35875): tablet 596da221dc99494c9d449bdc66c28fc6 (table test-table [id=2e3555e00de54316a4b4589a3a22e206]) successfully deleted
I20260806 10:53:22.546344 827 ts_tablet_manager.cc:1918] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:53:22.546628 827 log.cc:1196] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786013589554914-31592-0/minicluster-data/ts-0/wal/wals/596da221dc99494c9d449bdc66c28fc6
I20260806 10:53:22.547247 827 ts_tablet_manager.cc:1939] T 596da221dc99494c9d449bdc66c28fc6 P 44f59c51d5954d41bd6143fefdbbd813: Deleting consensus metadata
I20260806 10:53:22.548552 32734 catalog_manager.cc:4841] TS 44f59c51d5954d41bd6143fefdbbd813 (127.30.218.1:33557): tablet 596da221dc99494c9d449bdc66c28fc6 (table test-table [id=2e3555e00de54316a4b4589a3a22e206]) successfully deleted
Aug 06 10:53:22 dist-test-slave-49h3 krb5kdc[32702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013601, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.218.1@KRBTEST.COM
I20260806 10:53:22.566387 799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:22.523827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34772 (local address 127.30.218.1:33557)
0806 10:53:22.524014 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:22.524021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:22.548258 (+ 24237us) server_negotiation.cc:407] Connection header received
0806 10:53:22.548580 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:22.548586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:22.548705 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:22.548793 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:22.551088 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:22.551981 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:22.553878 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:22.554408 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:22.554527 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:22.559778 (+ 5251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:22.559792 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:22.559809 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:22.559865 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:22.563094 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:22.563954 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:22.563966 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:22.563968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:22.564153 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:22.564569 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:22.564573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:22.564575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:22.564991 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:22.565324 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:22.565738 (+ 414us) server_negotiation.cc:299] Negotiation successful
0806 10:53:22.566012 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260806 10:53:22.567103 490 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34772
I20260806 10:53:22.587428 840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:22.570436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52734 (local address 127.30.218.62:46855)
0806 10:53:22.570717 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:22.570723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:22.570829 (+ 106us) server_negotiation.cc:407] Connection header received
0806 10:53:22.570982 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:22.570989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:22.571081 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:22.571193 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:22.572561 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:22.573762 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:22.575047 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:22.575763 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:22.575903 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:22.578481 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:22.578489 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:22.578495 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:22.578549 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:22.582963 (+ 4414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:22.584019 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:22.584025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:22.584054 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:22.584227 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:22.584721 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:22.584727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:22.584731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:22.585146 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:22.585414 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:22.586767 (+ 1353us) server_negotiation.cc:299] Negotiation successful
0806 10:53:22.587049 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
W20260806 10:53:22.588145 32746 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52734
I20260806 10:53:22.590909 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 355
I20260806 10:53:22.612471 799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:22.593702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34776 (local address 127.30.218.1:33557)
0806 10:53:22.593874 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:22.593879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:22.594599 (+ 720us) server_negotiation.cc:407] Connection header received
0806 10:53:22.594778 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:22.594784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:22.594902 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:22.595014 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:22.596433 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:22.597805 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:22.599583 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:22.600252 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:22.600355 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:22.603727 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:22.603738 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:22.603743 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:22.603799 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:22.607919 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:22.609018 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:22.609024 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:22.609039 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:22.609250 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:22.609812 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:22.609818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:22.609822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:22.610235 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:22.610529 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:22.611824 (+ 1295us) server_negotiation.cc:299] Negotiation successful
0806 10:53:22.612110 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260806 10:53:22.613238 490 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34776
W20260806 10:53:22.614035 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 355: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:53:22.614267 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 355
I20260806 10:53:22.643648 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 499
Aug 06 10:53:22 dist-test-slave-49h3 krb5kdc[32702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013601, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.218.2@KRBTEST.COM
I20260806 10:53:22.660192 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:22.644584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55328 (local address 127.30.218.2:35875)
0806 10:53:22.645127 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:22.645131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:22.645147 (+ 16us) server_negotiation.cc:407] Connection header received
0806 10:53:22.645204 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:22.645208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:22.645285 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:22.645402 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:22.646790 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:22.647573 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:22.648914 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:22.649366 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:22.649538 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:22.654122 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:22.654141 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:22.654156 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:22.654207 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:22.656964 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:22.657962 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:22.657971 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:22.657973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:22.658154 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:22.658575 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:22.658579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:22.658581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:22.658946 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:22.659247 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:22.659588 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 10:53:22.659830 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":262,"threads_started":1}
W20260806 10:53:22.660917 633 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55328
W20260806 10:53:22.661573 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 499: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:53:22.661770 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 499
I20260806 10:53:22.680509 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 642
Aug 06 10:53:22 dist-test-slave-49h3 krb5kdc[32702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013601, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.218.3@KRBTEST.COM
I20260806 10:53:22.696869 862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:22.681344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60986 (local address 127.30.218.3:45145)
0806 10:53:22.681913 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:22.681917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:22.681933 (+ 16us) server_negotiation.cc:407] Connection header received
0806 10:53:22.681993 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:22.681997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:22.682069 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:22.682170 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:22.683517 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:22.684319 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:22.685670 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:22.686118 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:22.686238 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:22.690771 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:22.690790 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:22.690807 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:22.690853 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:22.693779 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:22.694599 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:22.694608 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:22.694610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:22.694778 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:22.695178 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:22.695182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:22.695184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:22.695571 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:22.695898 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:22.696204 (+ 306us) server_negotiation.cc:299] Negotiation successful
0806 10:53:22.696468 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":287,"threads_started":1}
W20260806 10:53:22.697610 778 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60986
W20260806 10:53:22.698292 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 642: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:53:22.698446 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 642
I20260806 10:53:22.716349 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 32715
I20260806 10:53:22.717558 323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:22.840270 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 32715
2026-08-06T10:53:22Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4770 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:53:22 dist-test-slave-49h3 krb5kdc[866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:53:22 dist-test-slave-49h3 krb5kdc[866](info): set up 2 sockets
Aug 06 10:53:22 dist-test-slave-49h3 krb5kdc[866](info): commencing operation
krb5kdc: starting...
W20260806 10:53:23.882908 31592 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:53:24 dist-test-slave-49h3 krb5kdc[866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:53:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:53:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:53:24.126546 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:42163
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:36913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:42163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:24.476114 880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:24.476486 880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:24.476624 880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:24.484419 880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:53:24.484571 880 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 10:53:24.484632 880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:24.484730 880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:24.484797 880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:53:24.484887 880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:53:24.495658 880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:36913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:42163
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:42163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:24.497186 880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:24.498687 880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:24.511618 885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:24.511593 886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:24.511535 888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:24.512679 880 server_base.cc:1034] running on GCE node
I20260806 10:53:24.513616 880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:24.515204 880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:24.516438 880 hybrid_clock.cc:648] HybridClock initialized: now 1786013604516361 us; error 43 us; skew 500 ppm
I20260806 10:53:24.516790 880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:24 dist-test-slave-49h3 krb5kdc[866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013604, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:24.520229 880 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:53:24.522025 880 webserver.cc:460] Webserver started at http://127.30.218.62:34859/ using document root <none> and password file <none>
I20260806 10:53:24.522851 880 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:24.522950 880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:24.523295 880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:24.526815 880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "085d238d8bf440d685ad93529dc3e736"
format_stamp: "Formatted at 2026-08-06 10:53:24 on dist-test-slave-49h3"
server_key: "0e3db37c297ba7be11a241ce0baee50c"
server_key_iv: "024ef57d79e5a4b5b8c2d26f369cf31d"
server_key_version: "encryptionkey@0"
I20260806 10:53:24.527740 880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "085d238d8bf440d685ad93529dc3e736"
format_stamp: "Formatted at 2026-08-06 10:53:24 on dist-test-slave-49h3"
server_key: "0e3db37c297ba7be11a241ce0baee50c"
server_key_iv: "024ef57d79e5a4b5b8c2d26f369cf31d"
server_key_version: "encryptionkey@0"
I20260806 10:53:24.533892 880 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260806 10:53:24.537973 896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:24.539587 880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:53:24.539788 880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "085d238d8bf440d685ad93529dc3e736"
format_stamp: "Formatted at 2026-08-06 10:53:24 on dist-test-slave-49h3"
server_key: "0e3db37c297ba7be11a241ce0baee50c"
server_key_iv: "024ef57d79e5a4b5b8c2d26f369cf31d"
server_key_version: "encryptionkey@0"
I20260806 10:53:24.540037 880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:24.586907 880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:24.591759 880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:24.592070 880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:24.614127 880 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:42163
I20260806 10:53:24.614135 959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:42163 every 8 connection(s)
I20260806 10:53:24.615651 880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:53:24.619829 961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:24.621985 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 880
I20260806 10:53:24.622226 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:53:24.623103 31592 external_mini_cluster.cc:1428] Setting key 2417995603518d943b886be42184cf26
I20260806 10:53:24.633138 961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736: Bootstrap starting.
I20260806 10:53:24.636567 961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:24.637755 961 log.cc:824] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736: Log is configured to *not* fsync() on all Append() calls
Aug 06 10:53:24 dist-test-slave-49h3 krb5kdc[866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:24.641633 961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736: No bootstrap required, opened a new log
I20260806 10:53:24.648617 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:24.625714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43972 (local address 127.30.218.62:42163)
0806 10:53:24.626670 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:24.626687 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:24.626735 (+ 48us) server_negotiation.cc:407] Connection header received
0806 10:53:24.628553 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:24.628579 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:24.629019 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:24.629440 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:24.631055 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:24.632426 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:24.633842 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:24.634586 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:24.634784 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:24.639107 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:24.639116 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:24.639134 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:24.639186 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:24.643169 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:24.644008 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:24.644022 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:24.644031 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:24.644260 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:24.644622 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:24.644628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:24.644631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:24.645437 (+ 806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:24.645805 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:24.646475 (+ 670us) server_negotiation.cc:299] Negotiation successful
0806 10:53:24.646996 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":441,"threads_started":1}
I20260806 10:53:24.652061 961 raft_consensus.cc:348] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085d238d8bf440d685ad93529dc3e736" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42163 } }
I20260806 10:53:24.652462 961 raft_consensus.cc:374] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:24.652540 961 raft_consensus.cc:729] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 085d238d8bf440d685ad93529dc3e736, State: Initialized, Role: FOLLOWER
I20260806 10:53:24.653095 961 consensus_queue.cc:260] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085d238d8bf440d685ad93529dc3e736" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42163 } }
I20260806 10:53:24.653324 961 raft_consensus.cc:388] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:53:24.653496 961 raft_consensus.cc:482] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:53:24.653656 961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:24.656101 961 raft_consensus.cc:504] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085d238d8bf440d685ad93529dc3e736" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42163 } }
I20260806 10:53:24.656628 961 leader_election.cc:302] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 085d238d8bf440d685ad93529dc3e736; no voters:
I20260806 10:53:24.657174 961 leader_election.cc:288] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:53:24.657315 966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:24.658113 966 raft_consensus.cc:686] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736 [term 1 LEADER]: Becoming Leader. State: Replica: 085d238d8bf440d685ad93529dc3e736, State: Running, Role: LEADER
I20260806 10:53:24.658711 966 consensus_queue.cc:237] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085d238d8bf440d685ad93529dc3e736" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42163 } }
I20260806 10:53:24.659478 961 sys_catalog.cc:564] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:53:24.661711 968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 085d238d8bf440d685ad93529dc3e736. Latest consensus state: current_term: 1 leader_uuid: "085d238d8bf440d685ad93529dc3e736" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085d238d8bf440d685ad93529dc3e736" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42163 } } }
I20260806 10:53:24.662504 968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736 [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:24.662127 967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "085d238d8bf440d685ad93529dc3e736" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085d238d8bf440d685ad93529dc3e736" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42163 } } }
I20260806 10:53:24.663012 967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736 [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:24.664180 974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:53:24.669209 974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:53:24.678941 974 catalog_manager.cc:1347] Generated new cluster ID: 47c71fe310594ed796db4735ed058afe
I20260806 10:53:24.679100 974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:53:24.690379 974 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:53:24.692498 974 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:53:24.705873 974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736: Generated new TSK 0
I20260806 10:53:24.706866 974 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
I20260806 10:53:24.770570 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:42163
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:36913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:25.119325 988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:25.119655 988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:25.119793 988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:25.127746 988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:25.127889 988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:25.128005 988 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 10:53:25.128072 988 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 10:53:25.128173 988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:53:25.138617 988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:36913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.218.62:42163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:25.140232 988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:25.141673 988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:25.155965 994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:25.156010 993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:25.155983 996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:25.156919 988 server_base.cc:1034] running on GCE node
I20260806 10:53:25.157620 988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:25.158694 988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:25.160063 988 hybrid_clock.cc:648] HybridClock initialized: now 1786013605159959 us; error 109 us; skew 500 ppm
I20260806 10:53:25.160449 988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:25 dist-test-slave-49h3 krb5kdc[866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013605, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:25.164058 988 init.cc:377] Logged in from keytab as kudu/127.30.218.1@KRBTEST.COM (short username kudu)
I20260806 10:53:25.165915 988 webserver.cc:460] Webserver started at http://127.30.218.1:40745/ using document root <none> and password file <none>
I20260806 10:53:25.166760 988 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:25.166893 988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:25.167232 988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:25.170531 988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "0bdc0ce86e3f46c6b6e579d883beabec"
format_stamp: "Formatted at 2026-08-06 10:53:25 on dist-test-slave-49h3"
server_key: "4860c26693b3524a2ee5a73bf9aba064"
server_key_iv: "2ca6546ebfa1cfa0af3a505631ce4df6"
server_key_version: "encryptionkey@0"
I20260806 10:53:25.171397 988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "0bdc0ce86e3f46c6b6e579d883beabec"
format_stamp: "Formatted at 2026-08-06 10:53:25 on dist-test-slave-49h3"
server_key: "4860c26693b3524a2ee5a73bf9aba064"
server_key_iv: "2ca6546ebfa1cfa0af3a505631ce4df6"
server_key_version: "encryptionkey@0"
I20260806 10:53:25.178172 988 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260806 10:53:25.182336 1003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:25.183841 988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:53:25.184069 988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "0bdc0ce86e3f46c6b6e579d883beabec"
format_stamp: "Formatted at 2026-08-06 10:53:25 on dist-test-slave-49h3"
server_key: "4860c26693b3524a2ee5a73bf9aba064"
server_key_iv: "2ca6546ebfa1cfa0af3a505631ce4df6"
server_key_version: "encryptionkey@0"
I20260806 10:53:25.184311 988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:25.204843 988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:25.209574 988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:25.209877 988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:25.212894 988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:25.213033 988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:25.213179 988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:25.213275 988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 10:53:25 dist-test-slave-49h3 krb5kdc[866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013605, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:25.240408 1013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:25.214227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42163 (local address 127.30.218.1:58985)
0806 10:53:25.215115 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:25.215159 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:25.217215 (+ 2056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:25.218081 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:25.218095 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:25.218986 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:25.219966 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:25.220001 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:25.221287 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:25.221293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:25.224056 (+ 2763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:25.224087 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:25.225377 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:25.225392 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:25.225827 (+ 435us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:25.226857 (+ 1030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:25.226894 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:25.230331 (+ 3437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:25.234279 (+ 3948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:25.234292 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:25.234319 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:25.235236 (+ 917us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:25.235842 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:25.235849 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:25.235853 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:25.236250 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:25.236974 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:25.236983 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:25.237552 (+ 569us) client_negotiation.cc:769] Sending connection context
0806 10:53:25.237899 (+ 347us) client_negotiation.cc:240] Negotiation successful
0806 10:53:25.238566 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":387,"threads_started":1}
I20260806 10:53:25.241681 1012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:25.213441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:58985 (local address 127.30.218.62:42163)
0806 10:53:25.214219 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:25.214224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:25.215304 (+ 1080us) server_negotiation.cc:407] Connection header received
0806 10:53:25.217580 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:25.217586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:25.217682 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:25.217783 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:25.220203 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:25.221092 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:25.224297 (+ 3205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:25.224875 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:25.225040 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:25.230662 (+ 5622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:25.230668 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:25.230672 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:25.230713 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:25.234049 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:25.235510 (+ 1461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:25.235517 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:25.235521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:25.235699 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:25.236399 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:25.236403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:25.236405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:25.236826 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:25.236981 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:25.241106 (+ 4125us) server_negotiation.cc:299] Negotiation successful
0806 10:53:25.241344 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":527,"threads_started":1}
I20260806 10:53:25.254886 988 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:34647
I20260806 10:53:25.254925 1126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:34647 every 8 connection(s)
I20260806 10:53:25.256604 988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:53:25.257758 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 988
I20260806 10:53:25.257965 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:53:25.258786 31592 external_mini_cluster.cc:1428] Setting key 624ae84cb999786004cf8d11d3818a4e
I20260806 10:53:25.259824 1127 heartbeater.cc:347] Connected to a master server at 127.30.218.62:42163
I20260806 10:53:25.260106 1127 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:25.261083 1127 heartbeater.cc:511] Master 127.30.218.62:42163 requested a full tablet report, sending...
I20260806 10:53:25.264017 913 ts_manager.cc:194] Registered new tserver with Master: 0bdc0ce86e3f46c6b6e579d883beabec (127.30.218.1:34647)
I20260806 10:53:25.264403 1114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:25.248824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42163 (local address 127.30.218.1:47403)
0806 10:53:25.249506 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:25.249531 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:25.249711 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:25.250341 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:25.250346 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:25.250975 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:25.251448 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:25.251464 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:25.252951 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:25.252956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:25.254212 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:25.254231 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:25.255046 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:25.255053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:25.255258 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:25.255928 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:25.255946 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:25.256819 (+ 873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:25.261169 (+ 4350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:25.261174 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:25.261179 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:25.261792 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:25.262272 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:25.262277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:25.262280 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:25.262440 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:25.263200 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:25.263205 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:25.263427 (+ 222us) client_negotiation.cc:769] Sending connection context
0806 10:53:25.263573 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 10:53:25.263954 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":350,"threads_started":1}
I20260806 10:53:25.265228 1012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:25.248739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:47403 (local address 127.30.218.62:42163)
0806 10:53:25.248920 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:25.248924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:25.249637 (+ 713us) server_negotiation.cc:407] Connection header received
0806 10:53:25.249910 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:25.249917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:25.250010 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:25.250086 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:25.251638 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:25.252793 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:25.254370 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:25.254829 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:25.254952 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:25.257046 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:25.257052 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:25.257055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:25.257092 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:25.260711 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:25.261953 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:25.261959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:25.261962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:25.262122 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:25.262564 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:25.262569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:25.262571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:25.262970 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:25.263118 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:25.264623 (+ 1505us) server_negotiation.cc:299] Negotiation successful
0806 10:53:25.264873 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 10:53:25.265893 913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.1@KRBTEST.COM'} at 127.30.218.1:58985
WARNING: no policy specified for kudu/127.30.218.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
I20260806 10:53:25.308532 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:42163
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:36913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:25.651669 1131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:25.651971 1131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:25.652134 1131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:25.659554 1131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:25.659727 1131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:25.659855 1131 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 10:53:25.659926 1131 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 10:53:25.660036 1131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:53:25.670879 1131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:36913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.218.62:42163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:25.672286 1131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:25.673647 1131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:25.686818 1136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:25.686792 1137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:25.686771 1139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:25.687566 1131 server_base.cc:1034] running on GCE node
I20260806 10:53:25.688318 1131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:25.689425 1131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:25.690685 1131 hybrid_clock.cc:648] HybridClock initialized: now 1786013605690630 us; error 45 us; skew 500 ppm
I20260806 10:53:25.691026 1131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:25 dist-test-slave-49h3 krb5kdc[866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013605, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:25.697355 1131 init.cc:377] Logged in from keytab as kudu/127.30.218.2@KRBTEST.COM (short username kudu)
I20260806 10:53:25.699072 1131 webserver.cc:460] Webserver started at http://127.30.218.2:35831/ using document root <none> and password file <none>
I20260806 10:53:25.699827 1131 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:25.699946 1131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:25.700284 1131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:25.703605 1131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "7ac2861735ad4be9b977108ebae1058d"
format_stamp: "Formatted at 2026-08-06 10:53:25 on dist-test-slave-49h3"
server_key: "360dc4f88027bc3104c32f463bd01a9c"
server_key_iv: "f135f8cb3bab959a8039997a61df0ecc"
server_key_version: "encryptionkey@0"
I20260806 10:53:25.704571 1131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "7ac2861735ad4be9b977108ebae1058d"
format_stamp: "Formatted at 2026-08-06 10:53:25 on dist-test-slave-49h3"
server_key: "360dc4f88027bc3104c32f463bd01a9c"
server_key_iv: "f135f8cb3bab959a8039997a61df0ecc"
server_key_version: "encryptionkey@0"
I20260806 10:53:25.710737 1131 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260806 10:53:25.714604 1146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:25.716039 1131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:53:25.716216 1131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "7ac2861735ad4be9b977108ebae1058d"
format_stamp: "Formatted at 2026-08-06 10:53:25 on dist-test-slave-49h3"
server_key: "360dc4f88027bc3104c32f463bd01a9c"
server_key_iv: "f135f8cb3bab959a8039997a61df0ecc"
server_key_version: "encryptionkey@0"
I20260806 10:53:25.716410 1131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:25.736487 1131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:25.740938 1131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:25.741247 1131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:25.744288 1131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:25.744429 1131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:25.744560 1131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:25.744659 1131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 10:53:25 dist-test-slave-49h3 krb5kdc[866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013605, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:25.769871 1012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:25.745061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:57201 (local address 127.30.218.62:42163)
0806 10:53:25.745241 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:25.745245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:25.747057 (+ 1812us) server_negotiation.cc:407] Connection header received
0806 10:53:25.749217 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:25.749223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:25.749323 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:25.749420 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:25.752406 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:25.753673 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:25.756140 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:25.756588 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:25.756711 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:25.762106 (+ 5395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:25.762112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:25.762115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:25.762157 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:25.765051 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:25.766273 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:25.766297 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:25.766301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:25.766496 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:25.767277 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:25.767283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:25.767287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:25.767739 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:25.767917 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:25.769075 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0806 10:53:25.769398 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 10:53:25.771034 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:25.746007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42163 (local address 127.30.218.2:57201)
0806 10:53:25.746873 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:25.746913 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:25.748853 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:25.749788 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:25.749803 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:25.751146 (+ 1343us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:25.752096 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:25.752147 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:25.753911 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:25.753917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:25.755921 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:25.755940 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:25.756865 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:25.756874 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:25.757112 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:25.757942 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:25.757974 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:25.761726 (+ 3752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:25.765297 (+ 3571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:25.765306 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:25.765324 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:25.766055 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:25.766685 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:25.766694 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:25.766699 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:25.767131 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:25.767899 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:25.767909 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:25.768518 (+ 609us) client_negotiation.cc:769] Sending connection context
0806 10:53:25.768896 (+ 378us) client_negotiation.cc:240] Negotiation successful
0806 10:53:25.769687 (+ 791us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":360,"threads_started":1}
I20260806 10:53:25.794821 1012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:25.777702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:54739 (local address 127.30.218.62:42163)
0806 10:53:25.777958 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:25.777966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:25.778587 (+ 621us) server_negotiation.cc:407] Connection header received
0806 10:53:25.778763 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:25.778769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:25.778871 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:25.778994 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:25.780034 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:25.781243 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:25.783355 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:25.783967 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:25.784116 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:25.787044 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:25.787053 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:25.787059 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:25.787111 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:25.791329 (+ 4218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:25.792224 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:25.792230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:25.792235 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:25.792485 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:25.792937 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:25.792943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:25.792947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:25.793490 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:25.793667 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:25.794213 (+ 546us) server_negotiation.cc:299] Negotiation successful
0806 10:53:25.794448 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 10:53:25.794978 1219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:25.777566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42163 (local address 127.30.218.2:54739)
0806 10:53:25.778454 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:25.778476 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:25.778635 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:25.779041 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:25.779046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:25.779566 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:25.779879 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:25.779890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:25.781433 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:25.781439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:25.783120 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:25.783146 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:25.784395 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:25.784400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:25.784654 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:25.785478 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:25.785508 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:25.786776 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:25.791552 (+ 4776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:25.791558 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:25.791561 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:25.792072 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:25.792642 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:25.792647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:25.792649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:25.792819 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:25.793636 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:25.793642 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:25.793900 (+ 258us) client_negotiation.cc:769] Sending connection context
0806 10:53:25.794048 (+ 148us) client_negotiation.cc:240] Negotiation successful
0806 10:53:25.794480 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":574,"threads_started":1}
I20260806 10:53:25.806382 1131 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:40095
I20260806 10:53:25.806416 1270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:40095 every 8 connection(s)
I20260806 10:53:25.807763 1131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:53:25.810102 1271 heartbeater.cc:347] Connected to a master server at 127.30.218.62:42163
I20260806 10:53:25.810367 1271 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:25.811110 1271 heartbeater.cc:511] Master 127.30.218.62:42163 requested a full tablet report, sending...
I20260806 10:53:25.812611 913 ts_manager.cc:194] Registered new tserver with Master: 7ac2861735ad4be9b977108ebae1058d (127.30.218.2:40095)
I20260806 10:53:25.813289 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 1131
I20260806 10:53:25.813503 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:53:25.813862 913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.2@KRBTEST.COM'} at 127.30.218.2:57201
I20260806 10:53:25.814087 31592 external_mini_cluster.cc:1428] Setting key 1c27eed2aa0d961b2ee9056c11fa30b6
WARNING: no policy specified for kudu/127.30.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
I20260806 10:53:25.862648 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:42163
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:36913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:26.212349 1275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:26.212689 1275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:26.212826 1275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:26.220631 1275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:26.220780 1275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:26.220912 1275 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 10:53:26.220976 1275 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 10:53:26.221063 1275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:53:26.231765 1275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:36913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.218.62:42163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:26.233224 1275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:26.234678 1275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:26.248454 1280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:26.248425 1281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:26.249059 1283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:26.249693 1275 server_base.cc:1034] running on GCE node
I20260806 10:53:26.250392 1275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:26.251448 1275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:26.252696 1275 hybrid_clock.cc:648] HybridClock initialized: now 1786013606252633 us; error 53 us; skew 500 ppm
I20260806 10:53:26.253062 1275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:26 dist-test-slave-49h3 krb5kdc[866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013606, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:26.257246 1275 init.cc:377] Logged in from keytab as kudu/127.30.218.3@KRBTEST.COM (short username kudu)
I20260806 10:53:26.259280 1275 webserver.cc:460] Webserver started at http://127.30.218.3:34327/ using document root <none> and password file <none>
I20260806 10:53:26.260268 1275 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:26.260421 1275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:26.260802 1275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:26.264333 1275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "aca3d6c624254657a57a0d6af3e2100d"
format_stamp: "Formatted at 2026-08-06 10:53:26 on dist-test-slave-49h3"
server_key: "bce1ad1b4c598e0a29930fd0a9952a1f"
server_key_iv: "1982551c638750cafdd18ad8b4388303"
server_key_version: "encryptionkey@0"
I20260806 10:53:26.265470 1275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "aca3d6c624254657a57a0d6af3e2100d"
format_stamp: "Formatted at 2026-08-06 10:53:26 on dist-test-slave-49h3"
server_key: "bce1ad1b4c598e0a29930fd0a9952a1f"
server_key_iv: "1982551c638750cafdd18ad8b4388303"
server_key_version: "encryptionkey@0"
I20260806 10:53:26.269923 1127 heartbeater.cc:503] Master 127.30.218.62:42163 was elected leader, sending a full tablet report...
I20260806 10:53:26.271935 1275 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 10:53:26.276149 1290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:26.277868 1275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 10:53:26.278048 1275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "aca3d6c624254657a57a0d6af3e2100d"
format_stamp: "Formatted at 2026-08-06 10:53:26 on dist-test-slave-49h3"
server_key: "bce1ad1b4c598e0a29930fd0a9952a1f"
server_key_iv: "1982551c638750cafdd18ad8b4388303"
server_key_version: "encryptionkey@0"
I20260806 10:53:26.278260 1275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:26.297256 1275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:26.302016 1275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:26.302295 1275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:26.305074 1275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:26.305183 1275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:26.305315 1275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:26.305408 1275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 10:53:26 dist-test-slave-49h3 krb5kdc[866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013606, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:26.332319 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:26.305967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:50743 (local address 127.30.218.62:42163)
0806 10:53:26.306523 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:26.306530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:26.307736 (+ 1206us) server_negotiation.cc:407] Connection header received
0806 10:53:26.310730 (+ 2994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:26.310735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:26.310815 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:26.310951 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:26.313800 (+ 2849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.314808 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.317528 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.318230 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.318394 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:26.323577 (+ 5183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:26.323586 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:26.323593 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:26.323649 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:26.327766 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:26.329066 (+ 1300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:26.329073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:26.329077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:26.329328 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:26.330100 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:26.330107 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:26.330111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:26.330548 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:26.330714 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:26.331610 (+ 896us) server_negotiation.cc:299] Negotiation successful
0806 10:53:26.331899 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":296,"threads_started":1}
I20260806 10:53:26.333561 1302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:26.306596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42163 (local address 127.30.218.3:50743)
0806 10:53:26.307516 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:26.307582 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:26.310347 (+ 2765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:26.311292 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:26.311310 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:26.312318 (+ 1008us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:26.313521 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:26.313562 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.315059 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.315063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:26.317274 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:26.317296 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.318566 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.318576 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:26.318911 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:26.319906 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:26.319940 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:26.323234 (+ 3294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:26.327987 (+ 4753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:26.328004 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:26.328028 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:26.328877 (+ 849us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:26.329487 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:26.329493 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:26.329497 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:26.329940 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:26.330696 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:26.330706 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:26.331158 (+ 452us) client_negotiation.cc:769] Sending connection context
0806 10:53:26.331441 (+ 283us) client_negotiation.cc:240] Negotiation successful
0806 10:53:26.332066 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":346,"threads_started":1}
I20260806 10:53:26.353430 1275 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:34833
I20260806 10:53:26.353533 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:34833 every 8 connection(s)
I20260806 10:53:26.354990 1275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:53:26.356714 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 1275
I20260806 10:53:26.356940 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:53:26.357564 31592 external_mini_cluster.cc:1428] Setting key 96cb87316673a42003b925fa83bf0035
I20260806 10:53:26.358669 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:26.340697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42163 (local address 127.30.218.3:36919)
0806 10:53:26.342008 (+ 1311us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:26.342037 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:26.342253 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:26.342684 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:26.342689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:26.343492 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:26.344086 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:26.344106 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.345468 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.345474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:26.347679 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:26.347708 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.349150 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.349158 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:26.349380 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:26.350283 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:26.350314 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:26.351655 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:26.355398 (+ 3743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:26.355405 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:26.355409 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:26.356102 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:26.356643 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:26.356648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:26.356652 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:26.356820 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:26.357587 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:26.357590 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:26.357796 (+ 206us) client_negotiation.cc:769] Sending connection context
0806 10:53:26.357912 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 10:53:26.358238 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1111,"thread_start_us":936,"threads_started":1}
I20260806 10:53:26.358701 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:26.340623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:36919 (local address 127.30.218.62:42163)
0806 10:53:26.340934 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:26.340941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:26.342173 (+ 1232us) server_negotiation.cc:407] Connection header received
0806 10:53:26.342416 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:26.342422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:26.342522 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:26.342633 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:26.344386 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.345262 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.347940 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.348607 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.348748 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:26.351920 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:26.351926 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:26.351930 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:26.351969 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:26.355238 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:26.356268 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:26.356274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:26.356277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:26.356481 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:26.356977 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:26.356983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:26.356987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:26.357422 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:26.357623 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:26.358082 (+ 459us) server_negotiation.cc:299] Negotiation successful
0806 10:53:26.358353 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260806 10:53:26.359133 1418 heartbeater.cc:347] Connected to a master server at 127.30.218.62:42163
I20260806 10:53:26.359366 1418 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:26.360252 1418 heartbeater.cc:511] Master 127.30.218.62:42163 requested a full tablet report, sending...
I20260806 10:53:26.361826 913 ts_manager.cc:194] Registered new tserver with Master: aca3d6c624254657a57a0d6af3e2100d (127.30.218.3:34833)
I20260806 10:53:26.362877 913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.3@KRBTEST.COM'} at 127.30.218.3:50743
I20260806 10:53:26.374396 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 10:53:26 dist-test-slave-49h3 krb5kdc[866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 10:53:26 dist-test-slave-49h3 krb5kdc[866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:26.432475 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:26.417507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41184 (local address 127.30.218.62:42163)
0806 10:53:26.417685 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:26.417690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:26.417914 (+ 224us) server_negotiation.cc:407] Connection header received
0806 10:53:26.418077 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:26.418081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:26.418168 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:26.418269 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:26.419860 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.420662 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.422078 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.422519 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.422636 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:26.426665 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:26.426671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:26.426675 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:26.426720 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:26.429612 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:26.430413 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:26.430418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:26.430420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:26.430578 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:26.430982 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:26.430986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:26.430988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:26.431299 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:26.431565 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:26.431883 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 10:53:26.432120 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 10:53:26.436995 913 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:53:26.444594 913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:53:26.470064 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:26.458785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:34647 (local address 127.0.0.1:58990)
0806 10:53:26.460921 (+ 2136us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:26.460956 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:26.461229 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:26.461972 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:26.461979 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:26.462019 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:53:26.462577 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:26.462592 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.464787 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.464795 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:26.466869 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:26.466886 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.468946 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.468953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:26.469189 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:26.469260 (+ 71us) client_negotiation.cc:769] Sending connection context
0806 10:53:26.469372 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 10:53:26.469634 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1933,"mutex_wait_us":34,"spinlock_wait_cycles":523520,"thread_start_us":2313,"threads_started":1}
I20260806 10:53:26.471184 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:26.459317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58990 (local address 127.30.218.1:34647)
0806 10:53:26.460530 (+ 1213us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:26.460539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:26.461103 (+ 564us) server_negotiation.cc:407] Connection header received
0806 10:53:26.461416 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:26.461424 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:26.461735 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:26.462817 (+ 1082us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:53:26.463019 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.464618 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.467087 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.468697 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.470351 (+ 1654us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:26.470585 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:26.470670 (+ 85us) server_negotiation.cc:299] Negotiation successful
0806 10:53:26.470862 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1052,"thread_start_us":921,"threads_started":1}
I20260806 10:53:26.471442 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:26.458610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37334 (local address 127.30.218.2:40095)
0806 10:53:26.460052 (+ 1442us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:26.460064 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:26.461176 (+ 1112us) server_negotiation.cc:407] Connection header received
0806 10:53:26.461502 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:26.461510 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:26.461766 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:26.461949 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:53:26.463391 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.464857 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.468991 (+ 4134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.470048 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.470186 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:26.470420 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:26.470961 (+ 541us) server_negotiation.cc:299] Negotiation successful
0806 10:53:26.471144 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1237,"thread_start_us":1072,"threads_started":1}
I20260806 10:53:26.471487 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:26.458419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:40095 (local address 127.0.0.1:37334)
0806 10:53:26.460923 (+ 2504us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:26.460957 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:26.461179 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:26.462004 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:26.462008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:26.462026 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:53:26.463145 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:26.463163 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.465024 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.465029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:26.468769 (+ 3740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:26.468783 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.470594 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.470601 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:26.470774 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:26.470801 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 10:53:26.470884 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 10:53:26.471027 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2248,"mutex_wait_us":421,"spinlock_wait_cycles":4992,"thread_start_us":2161,"threads_started":1}
I20260806 10:53:26.472164 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:26.462312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:34833 (local address 127.0.0.1:35460)
0806 10:53:26.463510 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:26.463526 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:26.463659 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:26.464593 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:26.464598 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:26.464624 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:53:26.465260 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:26.465275 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.466907 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.466916 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:26.469131 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:26.469146 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.471345 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.471351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:26.471542 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:26.471573 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 10:53:26.471666 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 10:53:26.471804 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1080,"mutex_wait_us":2,"thread_start_us":901,"threads_started":1}
I20260806 10:53:26.472395 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:26.461933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35460 (local address 127.30.218.3:34833)
0806 10:53:26.462854 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:26.462862 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:26.463631 (+ 769us) server_negotiation.cc:407] Connection header received
0806 10:53:26.463822 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:26.463829 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:26.464052 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:26.464272 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:53:26.465532 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.466767 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.469331 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.470820 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.470949 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:26.471181 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:26.471886 (+ 705us) server_negotiation.cc:299] Negotiation successful
0806 10:53:26.472062 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":532,"threads_started":1}
I20260806 10:53:26.475754 1198 tablet_service.cc:1467] Processing CreateTablet for tablet c57bb816e73c40c785215655319b1260 (DEFAULT_TABLE table=test-table [id=22e01a01cfee45199f6bb993386c49d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:26.475757 1055 tablet_service.cc:1467] Processing CreateTablet for tablet c57bb816e73c40c785215655319b1260 (DEFAULT_TABLE table=test-table [id=22e01a01cfee45199f6bb993386c49d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:26.475979 1343 tablet_service.cc:1467] Processing CreateTablet for tablet c57bb816e73c40c785215655319b1260 (DEFAULT_TABLE table=test-table [id=22e01a01cfee45199f6bb993386c49d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:26.477300 1055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c57bb816e73c40c785215655319b1260. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:26.477317 1198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c57bb816e73c40c785215655319b1260. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:26.477314 1343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c57bb816e73c40c785215655319b1260. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:26.491101 1438 tablet_bootstrap.cc:492] T c57bb816e73c40c785215655319b1260 P aca3d6c624254657a57a0d6af3e2100d: Bootstrap starting.
I20260806 10:53:26.495211 1438 tablet_bootstrap.cc:654] T c57bb816e73c40c785215655319b1260 P aca3d6c624254657a57a0d6af3e2100d: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:26.496366 1438 log.cc:824] T c57bb816e73c40c785215655319b1260 P aca3d6c624254657a57a0d6af3e2100d: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:26.499621 1438 tablet_bootstrap.cc:492] T c57bb816e73c40c785215655319b1260 P aca3d6c624254657a57a0d6af3e2100d: No bootstrap required, opened a new log
I20260806 10:53:26.499910 1438 ts_tablet_manager.cc:1397] T c57bb816e73c40c785215655319b1260 P aca3d6c624254657a57a0d6af3e2100d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260806 10:53:26.501713 1440 tablet_bootstrap.cc:492] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d: Bootstrap starting.
I20260806 10:53:26.502283 1441 tablet_bootstrap.cc:492] T c57bb816e73c40c785215655319b1260 P 0bdc0ce86e3f46c6b6e579d883beabec: Bootstrap starting.
I20260806 10:53:26.506107 1441 tablet_bootstrap.cc:654] T c57bb816e73c40c785215655319b1260 P 0bdc0ce86e3f46c6b6e579d883beabec: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:26.506781 1440 tablet_bootstrap.cc:654] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:26.507328 1441 log.cc:824] T c57bb816e73c40c785215655319b1260 P 0bdc0ce86e3f46c6b6e579d883beabec: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:26.508085 1440 log.cc:824] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:26.510842 1441 tablet_bootstrap.cc:492] T c57bb816e73c40c785215655319b1260 P 0bdc0ce86e3f46c6b6e579d883beabec: No bootstrap required, opened a new log
I20260806 10:53:26.510607 1438 raft_consensus.cc:348] T c57bb816e73c40c785215655319b1260 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:26.511091 1438 raft_consensus.cc:374] T c57bb816e73c40c785215655319b1260 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:26.511193 1438 raft_consensus.cc:729] T c57bb816e73c40c785215655319b1260 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aca3d6c624254657a57a0d6af3e2100d, State: Initialized, Role: FOLLOWER
I20260806 10:53:26.511407 1441 ts_tablet_manager.cc:1397] T c57bb816e73c40c785215655319b1260 P 0bdc0ce86e3f46c6b6e579d883beabec: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 10:53:26.512006 1438 consensus_queue.cc:260] T c57bb816e73c40c785215655319b1260 P aca3d6c624254657a57a0d6af3e2100d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:26.512421 1440 tablet_bootstrap.cc:492] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d: No bootstrap required, opened a new log
I20260806 10:53:26.512759 1440 ts_tablet_manager.cc:1397] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20260806 10:53:26.513368 1418 heartbeater.cc:503] Master 127.30.218.62:42163 was elected leader, sending a full tablet report...
I20260806 10:53:26.515837 1438 ts_tablet_manager.cc:1428] T c57bb816e73c40c785215655319b1260 P aca3d6c624254657a57a0d6af3e2100d: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260806 10:53:26.523809 1441 raft_consensus.cc:348] T c57bb816e73c40c785215655319b1260 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:26.523809 1440 raft_consensus.cc:348] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:26.524371 1440 raft_consensus.cc:374] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:26.524369 1441 raft_consensus.cc:374] T c57bb816e73c40c785215655319b1260 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:26.524485 1441 raft_consensus.cc:729] T c57bb816e73c40c785215655319b1260 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bdc0ce86e3f46c6b6e579d883beabec, State: Initialized, Role: FOLLOWER
I20260806 10:53:26.524485 1440 raft_consensus.cc:729] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac2861735ad4be9b977108ebae1058d, State: Initialized, Role: FOLLOWER
I20260806 10:53:26.525244 1440 consensus_queue.cc:260] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:26.525238 1441 consensus_queue.cc:260] T c57bb816e73c40c785215655319b1260 P 0bdc0ce86e3f46c6b6e579d883beabec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:26.526479 1271 heartbeater.cc:503] Master 127.30.218.62:42163 was elected leader, sending a full tablet report...
I20260806 10:53:26.527494 1440 ts_tablet_manager.cc:1428] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260806 10:53:26.527627 1441 ts_tablet_manager.cc:1428] T c57bb816e73c40c785215655319b1260 P 0bdc0ce86e3f46c6b6e579d883beabec: Time spent starting tablet: real 0.016s user 0.007s sys 0.008s
W20260806 10:53:26.559959 1272 tablet.cc:2369] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:53:26.606799 1419 tablet.cc:2369] T c57bb816e73c40c785215655319b1260 P aca3d6c624254657a57a0d6af3e2100d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:53:26.702872 1447 raft_consensus.cc:482] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:53:26.703264 1447 raft_consensus.cc:504] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:26.705042 1447 leader_election.cc:288] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bdc0ce86e3f46c6b6e579d883beabec (127.30.218.1:34647), aca3d6c624254657a57a0d6af3e2100d (127.30.218.3:34833)
I20260806 10:53:26.715268 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:26.705504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:40345 (local address 127.30.218.1:34647)
0806 10:53:26.705735 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:26.705741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:26.706577 (+ 836us) server_negotiation.cc:407] Connection header received
0806 10:53:26.706715 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:26.706721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:26.706805 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:26.706893 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:53:26.707797 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.709152 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.712563 (+ 3411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.713954 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.714168 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:26.714256 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:26.714811 (+ 555us) server_negotiation.cc:299] Negotiation successful
0806 10:53:26.714986 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 10:53:26.715268 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:26.705540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:45029 (local address 127.30.218.3:34833)
0806 10:53:26.705847 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:26.705853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:26.706748 (+ 895us) server_negotiation.cc:407] Connection header received
0806 10:53:26.706944 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:26.706949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:26.707031 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:26.707128 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:53:26.707969 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.709152 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.712563 (+ 3411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.713954 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.714168 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:26.714256 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:26.714819 (+ 563us) server_negotiation.cc:299] Negotiation successful
0806 10:53:26.714985 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260806 10:53:26.715633 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:26.705673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:34833 (local address 127.30.218.2:45029)
0806 10:53:26.706615 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:26.706633 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:26.706803 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:26.707215 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:26.707221 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:26.707249 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:53:26.707711 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:26.707721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.709381 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.709386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:26.712245 (+ 2859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:26.712270 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.714301 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.714309 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:26.714569 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:26.714645 (+ 76us) client_negotiation.cc:769] Sending connection context
0806 10:53:26.714758 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 10:53:26.715027 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":283,"thread_start_us":502,"threads_started":1}
I20260806 10:53:26.715883 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:26.705540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:34647 (local address 127.30.218.2:40345)
0806 10:53:26.706414 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:26.706433 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:26.706573 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:26.706957 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:26.706961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:26.706985 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:53:26.707498 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:26.707515 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.709379 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.709386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:26.712257 (+ 2871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:26.712270 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.714309 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.714314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:26.714592 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:26.714647 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 10:53:26.714758 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 10:53:26.715027 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":239,"thread_start_us":463,"threads_started":1}
I20260806 10:53:26.716753 1076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57bb816e73c40c785215655319b1260" candidate_uuid: "7ac2861735ad4be9b977108ebae1058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" is_pre_election: true
I20260806 10:53:26.716737 1363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57bb816e73c40c785215655319b1260" candidate_uuid: "7ac2861735ad4be9b977108ebae1058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aca3d6c624254657a57a0d6af3e2100d" is_pre_election: true
I20260806 10:53:26.717253 1076 raft_consensus.cc:2454] T c57bb816e73c40c785215655319b1260 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac2861735ad4be9b977108ebae1058d in term 0.
I20260806 10:53:26.717253 1363 raft_consensus.cc:2454] T c57bb816e73c40c785215655319b1260 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac2861735ad4be9b977108ebae1058d in term 0.
I20260806 10:53:26.717962 1147 leader_election.cc:302] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bdc0ce86e3f46c6b6e579d883beabec, 7ac2861735ad4be9b977108ebae1058d; no voters:
I20260806 10:53:26.718532 1447 raft_consensus.cc:2781] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:53:26.718757 1447 raft_consensus.cc:482] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:53:26.719019 1447 raft_consensus.cc:3037] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:26.721570 1447 raft_consensus.cc:504] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:26.722306 1447 leader_election.cc:288] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d [CANDIDATE]: Term 1 election: Requested vote from peers 0bdc0ce86e3f46c6b6e579d883beabec (127.30.218.1:34647), aca3d6c624254657a57a0d6af3e2100d (127.30.218.3:34833)
I20260806 10:53:26.722936 1363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57bb816e73c40c785215655319b1260" candidate_uuid: "7ac2861735ad4be9b977108ebae1058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aca3d6c624254657a57a0d6af3e2100d"
I20260806 10:53:26.722949 1076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57bb816e73c40c785215655319b1260" candidate_uuid: "7ac2861735ad4be9b977108ebae1058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bdc0ce86e3f46c6b6e579d883beabec"
I20260806 10:53:26.723162 1363 raft_consensus.cc:3037] T c57bb816e73c40c785215655319b1260 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:26.723165 1076 raft_consensus.cc:3037] T c57bb816e73c40c785215655319b1260 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:26.725987 1363 raft_consensus.cc:2454] T c57bb816e73c40c785215655319b1260 P aca3d6c624254657a57a0d6af3e2100d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac2861735ad4be9b977108ebae1058d in term 1.
I20260806 10:53:26.725987 1076 raft_consensus.cc:2454] T c57bb816e73c40c785215655319b1260 P 0bdc0ce86e3f46c6b6e579d883beabec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac2861735ad4be9b977108ebae1058d in term 1.
I20260806 10:53:26.726589 1147 leader_election.cc:302] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bdc0ce86e3f46c6b6e579d883beabec, 7ac2861735ad4be9b977108ebae1058d; no voters:
I20260806 10:53:26.726984 1447 raft_consensus.cc:2781] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:26.727586 1447 raft_consensus.cc:686] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d [term 1 LEADER]: Becoming Leader. State: Replica: 7ac2861735ad4be9b977108ebae1058d, State: Running, Role: LEADER
I20260806 10:53:26.728185 1447 consensus_queue.cc:237] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:26.734591 913 catalog_manager.cc:5495] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ac2861735ad4be9b977108ebae1058d (127.30.218.2). New cstate: current_term: 1 leader_uuid: "7ac2861735ad4be9b977108ebae1058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } health_report { overall_health: UNKNOWN } } }
W20260806 10:53:26.765841 1128 tablet.cc:2369] T c57bb816e73c40c785215655319b1260 P 0bdc0ce86e3f46c6b6e579d883beabec: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 10:53:26 dist-test-slave-49h3 krb5kdc[866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013604, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:26.816810 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:26.800532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41190 (local address 127.30.218.62:42163)
0806 10:53:26.800856 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:26.800864 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:26.801333 (+ 469us) server_negotiation.cc:407] Connection header received
0806 10:53:26.801525 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:26.801530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:26.801595 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:26.801692 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:26.803008 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.803836 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.805157 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.805665 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.805792 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:26.810647 (+ 4855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:26.810653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:26.810658 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:26.810713 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:26.813615 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:26.814594 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:26.814599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:26.814603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:26.814750 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:26.815131 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:26.815136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:26.815138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:26.815482 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:26.815648 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:26.816101 (+ 453us) server_negotiation.cc:299] Negotiation successful
0806 10:53:26.816477 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":157,"server-negotiator.queue_time_us":130}
I20260806 10:53:26.816811 1457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:26.800448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42163 (local address 127.0.0.1:41190)
0806 10:53:26.801215 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:26.801235 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:26.801385 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:26.801737 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:26.801740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:26.802478 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:26.802792 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:26.802804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.803983 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.803987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:26.805000 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:26.805014 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:26.805916 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:26.805924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:26.806100 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:26.806924 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:26.806950 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:26.810236 (+ 3286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:26.813801 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:26.813825 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:26.813833 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:26.814469 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:26.814878 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:26.814882 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:26.814884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:26.815024 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:26.815636 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:26.815646 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:26.815841 (+ 195us) client_negotiation.cc:769] Sending connection context
0806 10:53:26.815999 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 10:53:26.816324 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":424,"threads_started":1}
I20260806 10:53:26.822896 913 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:41190:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 10:53:26.824051 913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:53:26.831529 1343 tablet_service.cc:1467] Processing CreateTablet for tablet 72c239ae903e468e888dcc98e2f2b890 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=53b30058809e454685a842da22b2820a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 10:53:26.831488 1198 tablet_service.cc:1467] Processing CreateTablet for tablet 72c239ae903e468e888dcc98e2f2b890 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=53b30058809e454685a842da22b2820a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 10:53:26.831488 1055 tablet_service.cc:1467] Processing CreateTablet for tablet 72c239ae903e468e888dcc98e2f2b890 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=53b30058809e454685a842da22b2820a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 10:53:26.832444 1343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72c239ae903e468e888dcc98e2f2b890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:26.832458 1055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72c239ae903e468e888dcc98e2f2b890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:26.832453 1198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72c239ae903e468e888dcc98e2f2b890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:26.839218 1438 tablet_bootstrap.cc:492] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d: Bootstrap starting.
I20260806 10:53:26.840209 1440 tablet_bootstrap.cc:492] T 72c239ae903e468e888dcc98e2f2b890 P 7ac2861735ad4be9b977108ebae1058d: Bootstrap starting.
I20260806 10:53:26.840214 1441 tablet_bootstrap.cc:492] T 72c239ae903e468e888dcc98e2f2b890 P 0bdc0ce86e3f46c6b6e579d883beabec: Bootstrap starting.
I20260806 10:53:26.840986 1438 tablet_bootstrap.cc:654] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:26.841902 1440 tablet_bootstrap.cc:654] T 72c239ae903e468e888dcc98e2f2b890 P 7ac2861735ad4be9b977108ebae1058d: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:26.841902 1441 tablet_bootstrap.cc:654] T 72c239ae903e468e888dcc98e2f2b890 P 0bdc0ce86e3f46c6b6e579d883beabec: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:26.845283 1441 tablet_bootstrap.cc:492] T 72c239ae903e468e888dcc98e2f2b890 P 0bdc0ce86e3f46c6b6e579d883beabec: No bootstrap required, opened a new log
I20260806 10:53:26.845505 1441 ts_tablet_manager.cc:1397] T 72c239ae903e468e888dcc98e2f2b890 P 0bdc0ce86e3f46c6b6e579d883beabec: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 10:53:26.846061 1440 tablet_bootstrap.cc:492] T 72c239ae903e468e888dcc98e2f2b890 P 7ac2861735ad4be9b977108ebae1058d: No bootstrap required, opened a new log
I20260806 10:53:26.846069 1438 tablet_bootstrap.cc:492] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d: No bootstrap required, opened a new log
I20260806 10:53:26.846233 1440 ts_tablet_manager.cc:1397] T 72c239ae903e468e888dcc98e2f2b890 P 7ac2861735ad4be9b977108ebae1058d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 10:53:26.846231 1438 ts_tablet_manager.cc:1397] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260806 10:53:26.846283 1441 raft_consensus.cc:348] T 72c239ae903e468e888dcc98e2f2b890 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:26.846526 1441 raft_consensus.cc:374] T 72c239ae903e468e888dcc98e2f2b890 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:26.846625 1441 raft_consensus.cc:729] T 72c239ae903e468e888dcc98e2f2b890 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bdc0ce86e3f46c6b6e579d883beabec, State: Initialized, Role: FOLLOWER
I20260806 10:53:26.846882 1441 consensus_queue.cc:260] T 72c239ae903e468e888dcc98e2f2b890 P 0bdc0ce86e3f46c6b6e579d883beabec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:26.847057 1440 raft_consensus.cc:348] T 72c239ae903e468e888dcc98e2f2b890 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:26.847059 1438 raft_consensus.cc:348] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:26.847327 1440 raft_consensus.cc:374] T 72c239ae903e468e888dcc98e2f2b890 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:26.847338 1438 raft_consensus.cc:374] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:26.847429 1438 raft_consensus.cc:729] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aca3d6c624254657a57a0d6af3e2100d, State: Initialized, Role: FOLLOWER
I20260806 10:53:26.847429 1440 raft_consensus.cc:729] T 72c239ae903e468e888dcc98e2f2b890 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac2861735ad4be9b977108ebae1058d, State: Initialized, Role: FOLLOWER
I20260806 10:53:26.847438 1441 ts_tablet_manager.cc:1428] T 72c239ae903e468e888dcc98e2f2b890 P 0bdc0ce86e3f46c6b6e579d883beabec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 10:53:26.847669 1440 consensus_queue.cc:260] T 72c239ae903e468e888dcc98e2f2b890 P 7ac2861735ad4be9b977108ebae1058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:26.847718 1438 consensus_queue.cc:260] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:26.847826 1446 tablet_replica.cc:439] T 72c239ae903e468e888dcc98e2f2b890 P 0bdc0ce86e3f46c6b6e579d883beabec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } } }
I20260806 10:53:26.848234 1446 tablet_replica.cc:442] T 72c239ae903e468e888dcc98e2f2b890 P 0bdc0ce86e3f46c6b6e579d883beabec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:53:26.848243 1438 ts_tablet_manager.cc:1428] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260806 10:53:26.848569 1447 tablet_replica.cc:439] T 72c239ae903e468e888dcc98e2f2b890 P 7ac2861735ad4be9b977108ebae1058d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } } }
I20260806 10:53:26.848860 1440 ts_tablet_manager.cc:1428] T 72c239ae903e468e888dcc98e2f2b890 P 7ac2861735ad4be9b977108ebae1058d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 10:53:26.849045 1445 tablet_replica.cc:439] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } } }
I20260806 10:53:26.849357 1445 tablet_replica.cc:442] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:53:26.850639 1447 tablet_replica.cc:442] T 72c239ae903e468e888dcc98e2f2b890 P 7ac2861735ad4be9b977108ebae1058d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:53:27.111778 1445 raft_consensus.cc:482] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:53:27.111999 1445 raft_consensus.cc:504] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:27.113686 1445 leader_election.cc:288] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bdc0ce86e3f46c6b6e579d883beabec (127.30.218.1:34647), 7ac2861735ad4be9b977108ebae1058d (127.30.218.2:40095)
I20260806 10:53:27.121649 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:27.114207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:48129 (local address 127.30.218.1:34647)
0806 10:53:27.114442 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:27.114449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:27.115306 (+ 857us) server_negotiation.cc:407] Connection header received
0806 10:53:27.115418 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:27.115422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:27.115482 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:27.115553 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:53:27.116047 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.117052 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.119367 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.120601 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.120778 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:27.120859 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:27.121187 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 10:53:27.121335 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 10:53:27.121678 1462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:27.114066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:33197 (local address 127.30.218.2:40095)
0806 10:53:27.114966 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:27.114971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:27.114992 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:53:27.115069 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:27.115074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:27.115145 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:27.115231 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:53:27.115759 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.116789 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.119364 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.120601 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.120786 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:27.120858 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:27.121186 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 10:53:27.121332 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":610,"threads_started":1}
I20260806 10:53:27.121882 1461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:27.113984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:40095 (local address 127.30.218.3:33197)
0806 10:53:27.114654 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:27.114679 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:27.114843 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:27.115280 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:27.115285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:27.115375 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:53:27.115584 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:27.115596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.116922 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.116926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:27.119173 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:27.119196 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.120830 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.120835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:27.121044 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:27.121070 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 10:53:27.121158 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 10:53:27.121372 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":52,"spinlock_wait_cycles":24960,"thread_start_us":397,"threads_started":1}
I20260806 10:53:27.121907 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:27.113985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:34647 (local address 127.30.218.3:48129)
0806 10:53:27.115190 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:27.115206 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:27.115317 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:27.115590 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:27.115594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:27.115616 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:53:27.115909 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:27.115918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.117167 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.117172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:27.119243 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:27.119253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.120795 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.120801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:27.121011 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:27.121059 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 10:53:27.121158 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 10:53:27.121371 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1119,"mutex_wait_us":191,"thread_start_us":811,"threads_started":1}
I20260806 10:53:27.122756 1076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c239ae903e468e888dcc98e2f2b890" candidate_uuid: "aca3d6c624254657a57a0d6af3e2100d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" is_pre_election: true
I20260806 10:53:27.122917 1223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c239ae903e468e888dcc98e2f2b890" candidate_uuid: "aca3d6c624254657a57a0d6af3e2100d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac2861735ad4be9b977108ebae1058d" is_pre_election: true
I20260806 10:53:27.123059 1076 raft_consensus.cc:2454] T 72c239ae903e468e888dcc98e2f2b890 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aca3d6c624254657a57a0d6af3e2100d in term 0.
I20260806 10:53:27.123318 1223 raft_consensus.cc:2454] T 72c239ae903e468e888dcc98e2f2b890 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aca3d6c624254657a57a0d6af3e2100d in term 0.
I20260806 10:53:27.123572 1292 leader_election.cc:302] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bdc0ce86e3f46c6b6e579d883beabec, aca3d6c624254657a57a0d6af3e2100d; no voters:
I20260806 10:53:27.124018 1445 raft_consensus.cc:2781] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:53:27.124151 1445 raft_consensus.cc:482] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:53:27.124250 1445 raft_consensus.cc:3037] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:27.126435 1445 raft_consensus.cc:504] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:27.127038 1445 leader_election.cc:288] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d [CANDIDATE]: Term 1 election: Requested vote from peers 0bdc0ce86e3f46c6b6e579d883beabec (127.30.218.1:34647), 7ac2861735ad4be9b977108ebae1058d (127.30.218.2:40095)
I20260806 10:53:27.127542 1076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c239ae903e468e888dcc98e2f2b890" candidate_uuid: "aca3d6c624254657a57a0d6af3e2100d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bdc0ce86e3f46c6b6e579d883beabec"
I20260806 10:53:27.127575 1223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c239ae903e468e888dcc98e2f2b890" candidate_uuid: "aca3d6c624254657a57a0d6af3e2100d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac2861735ad4be9b977108ebae1058d"
I20260806 10:53:27.127776 1223 raft_consensus.cc:3037] T 72c239ae903e468e888dcc98e2f2b890 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:27.127794 1076 raft_consensus.cc:3037] T 72c239ae903e468e888dcc98e2f2b890 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:27.130880 1076 raft_consensus.cc:2454] T 72c239ae903e468e888dcc98e2f2b890 P 0bdc0ce86e3f46c6b6e579d883beabec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aca3d6c624254657a57a0d6af3e2100d in term 1.
I20260806 10:53:27.130894 1223 raft_consensus.cc:2454] T 72c239ae903e468e888dcc98e2f2b890 P 7ac2861735ad4be9b977108ebae1058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aca3d6c624254657a57a0d6af3e2100d in term 1.
I20260806 10:53:27.131422 1294 leader_election.cc:302] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ac2861735ad4be9b977108ebae1058d, aca3d6c624254657a57a0d6af3e2100d; no voters:
I20260806 10:53:27.131801 1445 raft_consensus.cc:2781] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:27.132555 1445 raft_consensus.cc:686] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d [term 1 LEADER]: Becoming Leader. State: Replica: aca3d6c624254657a57a0d6af3e2100d, State: Running, Role: LEADER
I20260806 10:53:27.133074 1445 consensus_queue.cc:237] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:27.136340 1464 tablet_replica.cc:439] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d: TxnStatusTablet state changed. Reason: New leader aca3d6c624254657a57a0d6af3e2100d. Latest consensus state: current_term: 1 leader_uuid: "aca3d6c624254657a57a0d6af3e2100d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } } }
I20260806 10:53:27.136724 1464 tablet_replica.cc:442] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d: This TxnStatusTablet replica's current role is: LEADER
I20260806 10:53:27.137470 1467 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 10:53:27.137728 1467 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 10:53:27.138114 912 catalog_manager.cc:5495] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d reported cstate change: term changed from 0 to 1, leader changed from <none> to aca3d6c624254657a57a0d6af3e2100d (127.30.218.3). New cstate: current_term: 1 leader_uuid: "aca3d6c624254657a57a0d6af3e2100d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } health_report { overall_health: HEALTHY } } }
I20260806 10:53:27.218740 1457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:27.212850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:34833 (local address 127.0.0.1:35462)
0806 10:53:27.213113 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:27.213132 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:27.213280 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:27.213789 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:27.213794 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:27.213807 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 10:53:27.213994 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:27.214003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.215164 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.215170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:27.216194 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:27.216207 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.216955 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.216961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:27.217122 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:27.217190 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 10:53:27.218144 (+ 954us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 10:53:27.218172 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 10:53:27.218258 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 10:53:27.218441 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 10:53:27.218748 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:27.212886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35462 (local address 127.30.218.3:34833)
0806 10:53:27.213079 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:27.213085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:27.213199 (+ 114us) server_negotiation.cc:407] Connection header received
0806 10:53:27.213523 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:27.213527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:27.213611 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:27.213738 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:27.214128 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.215027 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.216341 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.216785 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.216901 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:27.217383 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:27.218025 (+ 642us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:27.218176 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:27.218288 (+ 112us) server_negotiation.cc:299] Negotiation successful
0806 10:53:27.218388 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 10:53:27.227288 1223 raft_consensus.cc:1261] T 72c239ae903e468e888dcc98e2f2b890 P 7ac2861735ad4be9b977108ebae1058d [term 1 FOLLOWER]: Refusing update from remote peer aca3d6c624254657a57a0d6af3e2100d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:53:27.227478 1076 raft_consensus.cc:1261] T 72c239ae903e468e888dcc98e2f2b890 P 0bdc0ce86e3f46c6b6e579d883beabec [term 1 FOLLOWER]: Refusing update from remote peer aca3d6c624254657a57a0d6af3e2100d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:53:27.228324 1464 consensus_queue.cc:1035] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:53:27.228711 1445 consensus_queue.cc:1035] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:53:27.242537 1446 tablet_replica.cc:439] T 72c239ae903e468e888dcc98e2f2b890 P 0bdc0ce86e3f46c6b6e579d883beabec: TxnStatusTablet state changed. Reason: New leader aca3d6c624254657a57a0d6af3e2100d. Latest consensus state: current_term: 1 leader_uuid: "aca3d6c624254657a57a0d6af3e2100d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } } }
I20260806 10:53:27.242905 1446 tablet_replica.cc:442] T 72c239ae903e468e888dcc98e2f2b890 P 0bdc0ce86e3f46c6b6e579d883beabec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:53:27.241300 1447 tablet_replica.cc:439] T 72c239ae903e468e888dcc98e2f2b890 P 7ac2861735ad4be9b977108ebae1058d: TxnStatusTablet state changed. Reason: New leader aca3d6c624254657a57a0d6af3e2100d. Latest consensus state: current_term: 1 leader_uuid: "aca3d6c624254657a57a0d6af3e2100d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } } }
I20260806 10:53:27.244047 1447 tablet_replica.cc:442] T 72c239ae903e468e888dcc98e2f2b890 P 7ac2861735ad4be9b977108ebae1058d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:53:27.246682 1470 mvcc.cc:205] Tried to move back new op lower bound from 7315511735192403968 to 7315511734820855808. Current Snapshot: MvccSnapshot[applied={T|T < 7315511735192403968}]
I20260806 10:53:27.261281 1446 tablet_replica.cc:439] T 72c239ae903e468e888dcc98e2f2b890 P 0bdc0ce86e3f46c6b6e579d883beabec: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aca3d6c624254657a57a0d6af3e2100d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } } }
I20260806 10:53:27.261615 1446 tablet_replica.cc:442] T 72c239ae903e468e888dcc98e2f2b890 P 0bdc0ce86e3f46c6b6e579d883beabec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:53:27.267962 1445 tablet_replica.cc:439] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aca3d6c624254657a57a0d6af3e2100d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } } }
I20260806 10:53:27.268337 1445 tablet_replica.cc:442] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d: This TxnStatusTablet replica's current role is: LEADER
I20260806 10:53:27.275812 1447 tablet_replica.cc:439] T 72c239ae903e468e888dcc98e2f2b890 P 7ac2861735ad4be9b977108ebae1058d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aca3d6c624254657a57a0d6af3e2100d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } } }
I20260806 10:53:27.276152 1447 tablet_replica.cc:442] T 72c239ae903e468e888dcc98e2f2b890 P 7ac2861735ad4be9b977108ebae1058d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:53:27.281185 1447 consensus_queue.cc:1035] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:53:27.315634 1464 tablet_replica.cc:439] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aca3d6c624254657a57a0d6af3e2100d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } } }
I20260806 10:53:27.315992 1464 tablet_replica.cc:442] T 72c239ae903e468e888dcc98e2f2b890 P aca3d6c624254657a57a0d6af3e2100d: This TxnStatusTablet replica's current role is: LEADER
I20260806 10:53:27.319583 1479 consensus_queue.cc:1035] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:53:27.320039 1462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:27.295182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37344 (local address 127.30.218.2:40095)
0806 10:53:27.295422 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:27.295429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:27.312859 (+ 17430us) server_negotiation.cc:407] Connection header received
0806 10:53:27.313142 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:27.313149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:27.313309 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:27.313554 (+ 245us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:27.314321 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.315345 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.317063 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.317692 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.317823 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:27.318449 (+ 626us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:27.319238 (+ 789us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:27.319368 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:27.319633 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 10:53:27.319743 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 10:53:27.341965 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:27.335927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:49779 (local address 127.30.218.3:34833)
0806 10:53:27.336134 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:27.336141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:27.336630 (+ 489us) server_negotiation.cc:407] Connection header received
0806 10:53:27.336842 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:27.336846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:27.336914 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:27.336991 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:27.337519 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.338273 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.339651 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.340402 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.340556 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:27.341258 (+ 702us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:27.341383 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:27.341512 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:27.341649 (+ 137us) server_negotiation.cc:299] Negotiation successful
0806 10:53:27.341726 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 10:53:27.342429 1487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:27.335844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:34833 (local address 127.30.218.2:49779)
0806 10:53:27.336521 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:27.336538 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:27.336695 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:27.337030 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:27.337035 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:27.337047 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 10:53:27.337339 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:27.337348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.338405 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.338410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:27.339506 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:27.339520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.340676 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.340683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:27.340891 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:27.340985 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 10:53:27.341530 (+ 545us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 10:53:27.341544 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 10:53:27.341631 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 10:53:27.341806 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":364,"threads_started":1}
I20260806 10:53:27.393029 1462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:27.383690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:47379 (local address 127.30.218.2:40095)
0806 10:53:27.383882 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:27.383889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:27.386154 (+ 2265us) server_negotiation.cc:407] Connection header received
0806 10:53:27.386343 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:27.386349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:27.386465 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:27.386563 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:27.387257 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.388441 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.391220 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.391635 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.391756 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:27.392338 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:27.392453 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:27.392522 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:27.392723 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 10:53:27.392806 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 10:53:27.393384 1491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:27.383695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:40095 (local address 127.30.218.3:47379)
0806 10:53:27.385981 (+ 2286us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:27.386004 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:27.386191 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:27.386612 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:27.386618 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:27.386632 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 10:53:27.387073 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:27.387086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.388926 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.388932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:27.391030 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:27.391049 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.391842 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.391850 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:27.392059 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:27.392139 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 10:53:27.392555 (+ 416us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 10:53:27.392568 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 10:53:27.392652 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 10:53:27.392825 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2064,"thread_start_us":1891,"threads_started":1}
I20260806 10:53:27.441402 912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:53:27.441804 912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:53:27.446795 912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736: Sending DeleteTablet for 3 replicas of tablet c57bb816e73c40c785215655319b1260
I20260806 10:53:27.448423 1198 tablet_service.cc:1514] Processing DeleteTablet for tablet c57bb816e73c40c785215655319b1260 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:53:27 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:37334
I20260806 10:53:27.449642 1343 tablet_service.cc:1514] Processing DeleteTablet for tablet c57bb816e73c40c785215655319b1260 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:53:27 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:35460
I20260806 10:53:27.450443 1055 tablet_service.cc:1514] Processing DeleteTablet for tablet c57bb816e73c40c785215655319b1260 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:53:27 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:58990
I20260806 10:53:27.451766 912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:53:27.453598 912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:53:27.465350 1197 tablet_service.cc:1467] Processing CreateTablet for tablet 3d540a09c04f459cb2247751a7e6df57 (DEFAULT_TABLE table=test-table [id=60a5ae58cdd7402aacfef62bb54c3335]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:27.465672 1342 tablet_service.cc:1467] Processing CreateTablet for tablet 3d540a09c04f459cb2247751a7e6df57 (DEFAULT_TABLE table=test-table [id=60a5ae58cdd7402aacfef62bb54c3335]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:27.466075 1197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d540a09c04f459cb2247751a7e6df57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:27.466286 1342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d540a09c04f459cb2247751a7e6df57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:27.466987 1054 tablet_service.cc:1467] Processing CreateTablet for tablet 3d540a09c04f459cb2247751a7e6df57 (DEFAULT_TABLE table=test-table [id=60a5ae58cdd7402aacfef62bb54c3335]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:27.467475 1054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d540a09c04f459cb2247751a7e6df57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:27.478876 1494 tablet_replica.cc:330] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d: stopping tablet replica
I20260806 10:53:27.479489 1494 raft_consensus.cc:2229] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:53:27.481848 1494 raft_consensus.cc:2258] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:53:27.484119 1494 ts_tablet_manager.cc:1905] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:53:27.485785 1498 tablet_bootstrap.cc:492] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d: Bootstrap starting.
I20260806 10:53:27.486243 1496 tablet_replica.cc:330] T c57bb816e73c40c785215655319b1260 P 0bdc0ce86e3f46c6b6e579d883beabec: stopping tablet replica
I20260806 10:53:27.487176 1496 raft_consensus.cc:2229] T c57bb816e73c40c785215655319b1260 P 0bdc0ce86e3f46c6b6e579d883beabec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:53:27.487702 1496 raft_consensus.cc:2258] T c57bb816e73c40c785215655319b1260 P 0bdc0ce86e3f46c6b6e579d883beabec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:53:27.486652 1499 tablet_bootstrap.cc:492] T 3d540a09c04f459cb2247751a7e6df57 P 0bdc0ce86e3f46c6b6e579d883beabec: Bootstrap starting.
I20260806 10:53:27.490653 1499 tablet_bootstrap.cc:654] T 3d540a09c04f459cb2247751a7e6df57 P 0bdc0ce86e3f46c6b6e579d883beabec: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:27.495195 1497 tablet_bootstrap.cc:492] T 3d540a09c04f459cb2247751a7e6df57 P 7ac2861735ad4be9b977108ebae1058d: Bootstrap starting.
I20260806 10:53:27.493005 1496 ts_tablet_manager.cc:1905] T c57bb816e73c40c785215655319b1260 P 0bdc0ce86e3f46c6b6e579d883beabec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:53:27.497371 1498 tablet_bootstrap.cc:654] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:27.498638 1495 tablet_replica.cc:330] T c57bb816e73c40c785215655319b1260 P aca3d6c624254657a57a0d6af3e2100d: stopping tablet replica
I20260806 10:53:27.499153 1495 raft_consensus.cc:2229] T c57bb816e73c40c785215655319b1260 P aca3d6c624254657a57a0d6af3e2100d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:53:27.499538 1495 raft_consensus.cc:2258] T c57bb816e73c40c785215655319b1260 P aca3d6c624254657a57a0d6af3e2100d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:53:27.500118 1497 tablet_bootstrap.cc:654] T 3d540a09c04f459cb2247751a7e6df57 P 7ac2861735ad4be9b977108ebae1058d: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:27.501691 1495 ts_tablet_manager.cc:1905] T c57bb816e73c40c785215655319b1260 P aca3d6c624254657a57a0d6af3e2100d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:53:27.502360 1494 ts_tablet_manager.cc:1918] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:53:27.502629 1494 log.cc:1196] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/c57bb816e73c40c785215655319b1260
I20260806 10:53:27.503330 1494 ts_tablet_manager.cc:1939] T c57bb816e73c40c785215655319b1260 P 7ac2861735ad4be9b977108ebae1058d: Deleting consensus metadata
I20260806 10:53:27.507198 899 catalog_manager.cc:4841] TS 7ac2861735ad4be9b977108ebae1058d (127.30.218.2:40095): tablet c57bb816e73c40c785215655319b1260 (table test-table [id=22e01a01cfee45199f6bb993386c49d9]) successfully deleted
I20260806 10:53:27.510286 1499 tablet_bootstrap.cc:492] T 3d540a09c04f459cb2247751a7e6df57 P 0bdc0ce86e3f46c6b6e579d883beabec: No bootstrap required, opened a new log
I20260806 10:53:27.510584 1499 ts_tablet_manager.cc:1397] T 3d540a09c04f459cb2247751a7e6df57 P 0bdc0ce86e3f46c6b6e579d883beabec: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.003s
I20260806 10:53:27.511566 1499 raft_consensus.cc:348] T 3d540a09c04f459cb2247751a7e6df57 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:27.511958 1499 raft_consensus.cc:374] T 3d540a09c04f459cb2247751a7e6df57 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:27.512099 1499 raft_consensus.cc:729] T 3d540a09c04f459cb2247751a7e6df57 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bdc0ce86e3f46c6b6e579d883beabec, State: Initialized, Role: FOLLOWER
I20260806 10:53:27.512467 1499 consensus_queue.cc:260] T 3d540a09c04f459cb2247751a7e6df57 P 0bdc0ce86e3f46c6b6e579d883beabec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:27.513285 1499 ts_tablet_manager.cc:1428] T 3d540a09c04f459cb2247751a7e6df57 P 0bdc0ce86e3f46c6b6e579d883beabec: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260806 10:53:27.515146 1497 tablet_bootstrap.cc:492] T 3d540a09c04f459cb2247751a7e6df57 P 7ac2861735ad4be9b977108ebae1058d: No bootstrap required, opened a new log
I20260806 10:53:27.515380 1497 ts_tablet_manager.cc:1397] T 3d540a09c04f459cb2247751a7e6df57 P 7ac2861735ad4be9b977108ebae1058d: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20260806 10:53:27.516691 1497 raft_consensus.cc:348] T 3d540a09c04f459cb2247751a7e6df57 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:27.517046 1497 raft_consensus.cc:374] T 3d540a09c04f459cb2247751a7e6df57 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:27.517191 1497 raft_consensus.cc:729] T 3d540a09c04f459cb2247751a7e6df57 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac2861735ad4be9b977108ebae1058d, State: Initialized, Role: FOLLOWER
I20260806 10:53:27.517843 1497 consensus_queue.cc:260] T 3d540a09c04f459cb2247751a7e6df57 P 7ac2861735ad4be9b977108ebae1058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:27.518878 1497 ts_tablet_manager.cc:1428] T 3d540a09c04f459cb2247751a7e6df57 P 7ac2861735ad4be9b977108ebae1058d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260806 10:53:27.521991 1498 tablet_bootstrap.cc:492] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d: No bootstrap required, opened a new log
I20260806 10:53:27.522202 1498 ts_tablet_manager.cc:1397] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.004s
I20260806 10:53:27.523190 1498 raft_consensus.cc:348] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:27.523491 1498 raft_consensus.cc:374] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:27.523603 1498 raft_consensus.cc:729] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aca3d6c624254657a57a0d6af3e2100d, State: Initialized, Role: FOLLOWER
I20260806 10:53:27.523866 1498 consensus_queue.cc:260] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:27.524542 1498 ts_tablet_manager.cc:1428] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260806 10:53:27.526621 1495 ts_tablet_manager.cc:1918] T c57bb816e73c40c785215655319b1260 P aca3d6c624254657a57a0d6af3e2100d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:53:27.526814 1495 log.cc:1196] T c57bb816e73c40c785215655319b1260 P aca3d6c624254657a57a0d6af3e2100d: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/c57bb816e73c40c785215655319b1260
I20260806 10:53:27.527407 1495 ts_tablet_manager.cc:1939] T c57bb816e73c40c785215655319b1260 P aca3d6c624254657a57a0d6af3e2100d: Deleting consensus metadata
I20260806 10:53:27.523366 1496 ts_tablet_manager.cc:1918] T c57bb816e73c40c785215655319b1260 P 0bdc0ce86e3f46c6b6e579d883beabec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:53:27.528224 1496 log.cc:1196] T c57bb816e73c40c785215655319b1260 P 0bdc0ce86e3f46c6b6e579d883beabec: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-0/wal/wals/c57bb816e73c40c785215655319b1260
I20260806 10:53:27.528772 1496 ts_tablet_manager.cc:1939] T c57bb816e73c40c785215655319b1260 P 0bdc0ce86e3f46c6b6e579d883beabec: Deleting consensus metadata
I20260806 10:53:27.531000 897 catalog_manager.cc:4841] TS 0bdc0ce86e3f46c6b6e579d883beabec (127.30.218.1:34647): tablet c57bb816e73c40c785215655319b1260 (table test-table [id=22e01a01cfee45199f6bb993386c49d9]) successfully deleted
I20260806 10:53:27.532291 899 catalog_manager.cc:4841] TS aca3d6c624254657a57a0d6af3e2100d (127.30.218.3:34833): tablet c57bb816e73c40c785215655319b1260 (table test-table [id=22e01a01cfee45199f6bb993386c49d9]) successfully deleted
I20260806 10:53:27.578254 1464 raft_consensus.cc:482] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:53:27.578521 1464 raft_consensus.cc:504] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:27.579325 1464 leader_election.cc:288] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bdc0ce86e3f46c6b6e579d883beabec (127.30.218.1:34647), 7ac2861735ad4be9b977108ebae1058d (127.30.218.2:40095)
I20260806 10:53:27.580000 1076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d540a09c04f459cb2247751a7e6df57" candidate_uuid: "aca3d6c624254657a57a0d6af3e2100d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" is_pre_election: true
I20260806 10:53:27.580183 1223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d540a09c04f459cb2247751a7e6df57" candidate_uuid: "aca3d6c624254657a57a0d6af3e2100d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac2861735ad4be9b977108ebae1058d" is_pre_election: true
I20260806 10:53:27.580310 1076 raft_consensus.cc:2454] T 3d540a09c04f459cb2247751a7e6df57 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aca3d6c624254657a57a0d6af3e2100d in term 0.
I20260806 10:53:27.580478 1223 raft_consensus.cc:2454] T 3d540a09c04f459cb2247751a7e6df57 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aca3d6c624254657a57a0d6af3e2100d in term 0.
I20260806 10:53:27.581018 1292 leader_election.cc:302] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bdc0ce86e3f46c6b6e579d883beabec, aca3d6c624254657a57a0d6af3e2100d; no voters:
I20260806 10:53:27.581467 1464 raft_consensus.cc:2781] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:53:27.581591 1464 raft_consensus.cc:482] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:53:27.581718 1464 raft_consensus.cc:3037] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:27.585414 1464 raft_consensus.cc:504] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:27.586936 1076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d540a09c04f459cb2247751a7e6df57" candidate_uuid: "aca3d6c624254657a57a0d6af3e2100d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bdc0ce86e3f46c6b6e579d883beabec"
I20260806 10:53:27.586977 1223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d540a09c04f459cb2247751a7e6df57" candidate_uuid: "aca3d6c624254657a57a0d6af3e2100d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac2861735ad4be9b977108ebae1058d"
I20260806 10:53:27.587226 1076 raft_consensus.cc:3037] T 3d540a09c04f459cb2247751a7e6df57 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:27.587267 1223 raft_consensus.cc:3037] T 3d540a09c04f459cb2247751a7e6df57 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:27.590457 1076 raft_consensus.cc:2454] T 3d540a09c04f459cb2247751a7e6df57 P 0bdc0ce86e3f46c6b6e579d883beabec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aca3d6c624254657a57a0d6af3e2100d in term 1.
I20260806 10:53:27.590679 1223 raft_consensus.cc:2454] T 3d540a09c04f459cb2247751a7e6df57 P 7ac2861735ad4be9b977108ebae1058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aca3d6c624254657a57a0d6af3e2100d in term 1.
I20260806 10:53:27.591310 1292 leader_election.cc:302] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bdc0ce86e3f46c6b6e579d883beabec, aca3d6c624254657a57a0d6af3e2100d; no voters:
I20260806 10:53:27.591464 1464 leader_election.cc:288] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d [CANDIDATE]: Term 1 election: Requested vote from peers 0bdc0ce86e3f46c6b6e579d883beabec (127.30.218.1:34647), 7ac2861735ad4be9b977108ebae1058d (127.30.218.2:40095)
I20260806 10:53:27.591722 1445 raft_consensus.cc:2781] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:27.591882 1445 raft_consensus.cc:686] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d [term 1 LEADER]: Becoming Leader. State: Replica: aca3d6c624254657a57a0d6af3e2100d, State: Running, Role: LEADER
I20260806 10:53:27.592211 1445 consensus_queue.cc:237] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:27.595657 911 catalog_manager.cc:5495] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d reported cstate change: term changed from 0 to 1, leader changed from <none> to aca3d6c624254657a57a0d6af3e2100d (127.30.218.3). New cstate: current_term: 1 leader_uuid: "aca3d6c624254657a57a0d6af3e2100d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } health_report { overall_health: HEALTHY } } }
I20260806 10:53:27.631258 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:27.624094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.30.218.3:34833)
0806 10:53:27.624276 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:27.624281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:27.624300 (+ 19us) server_negotiation.cc:407] Connection header received
0806 10:53:27.624368 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:27.624373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:27.624458 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:27.624542 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:27.625105 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.626329 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.628356 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.629096 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.629215 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:27.629892 (+ 677us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:27.630088 (+ 196us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:27.630174 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:27.630794 (+ 620us) server_negotiation.cc:299] Negotiation successful
0806 10:53:27.630894 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 10:53:27.656864 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:27.650561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:37151 (local address 127.30.218.3:34833)
0806 10:53:27.650731 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:27.650737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:27.650758 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:53:27.650821 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:27.650826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:27.650910 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:27.651009 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:27.651429 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.652704 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.654670 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.655340 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.655471 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:27.655982 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:27.656144 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:27.656244 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:27.656432 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 10:53:27.656523 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 10:53:27.656941 1491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:27.650312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:34833 (local address 127.30.218.3:37151)
0806 10:53:27.650551 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:27.650574 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:27.650716 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:27.651043 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:27.651049 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:27.651063 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 10:53:27.651287 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:27.651299 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.652910 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.652916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:27.654490 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:27.654507 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:27.655548 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:27.655554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:27.655722 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:27.655779 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 10:53:27.656265 (+ 486us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 10:53:27.656279 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 10:53:27.656382 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 10:53:27.656570 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":10}
I20260806 10:53:27.670009 1222 raft_consensus.cc:1261] T 3d540a09c04f459cb2247751a7e6df57 P 7ac2861735ad4be9b977108ebae1058d [term 1 FOLLOWER]: Refusing update from remote peer aca3d6c624254657a57a0d6af3e2100d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:53:27.670914 1478 consensus_queue.cc:1035] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:53:27.692863 1076 raft_consensus.cc:1261] T 3d540a09c04f459cb2247751a7e6df57 P 0bdc0ce86e3f46c6b6e579d883beabec [term 1 FOLLOWER]: Refusing update from remote peer aca3d6c624254657a57a0d6af3e2100d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:53:27.693599 1506 consensus_queue.cc:1035] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:53:27.697150 1471 mvcc.cc:205] Tried to move back new op lower bound from 7315511737002782720 to 7315511736697950208. Current Snapshot: MvccSnapshot[applied={T|T < 7315511737002782720}]
W20260806 10:53:27.719178 1470 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 10:53:29.332442 911 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:53:29.332733 911 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:53:29.337394 911 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736: Sending DeleteTablet for 3 replicas of tablet 3d540a09c04f459cb2247751a7e6df57
I20260806 10:53:29.338464 1343 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d540a09c04f459cb2247751a7e6df57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:53:29 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:35460
I20260806 10:53:29.338512 1054 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d540a09c04f459cb2247751a7e6df57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:53:29 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:58990
I20260806 10:53:29.338591 1197 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d540a09c04f459cb2247751a7e6df57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:53:29 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:37334
I20260806 10:53:29.348629 1533 tablet_replica.cc:330] T 3d540a09c04f459cb2247751a7e6df57 P 7ac2861735ad4be9b977108ebae1058d: stopping tablet replica
I20260806 10:53:29.348869 1532 tablet_replica.cc:330] T 3d540a09c04f459cb2247751a7e6df57 P 0bdc0ce86e3f46c6b6e579d883beabec: stopping tablet replica
I20260806 10:53:29.349030 1533 raft_consensus.cc:2229] T 3d540a09c04f459cb2247751a7e6df57 P 7ac2861735ad4be9b977108ebae1058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:53:29.349102 1532 raft_consensus.cc:2229] T 3d540a09c04f459cb2247751a7e6df57 P 0bdc0ce86e3f46c6b6e579d883beabec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:53:29.349388 1532 raft_consensus.cc:2258] T 3d540a09c04f459cb2247751a7e6df57 P 0bdc0ce86e3f46c6b6e579d883beabec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:53:29.349414 1533 raft_consensus.cc:2258] T 3d540a09c04f459cb2247751a7e6df57 P 7ac2861735ad4be9b977108ebae1058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:53:29.350021 1531 tablet_replica.cc:330] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d: stopping tablet replica
I20260806 10:53:29.350242 1531 raft_consensus.cc:2229] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:53:29.350904 1531 raft_consensus.cc:2258] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:53:29.350919 1533 ts_tablet_manager.cc:1905] T 3d540a09c04f459cb2247751a7e6df57 P 7ac2861735ad4be9b977108ebae1058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:53:29.350919 1532 ts_tablet_manager.cc:1905] T 3d540a09c04f459cb2247751a7e6df57 P 0bdc0ce86e3f46c6b6e579d883beabec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:53:29.351964 1531 ts_tablet_manager.cc:1905] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:53:29.358551 1532 ts_tablet_manager.cc:1918] T 3d540a09c04f459cb2247751a7e6df57 P 0bdc0ce86e3f46c6b6e579d883beabec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 10:53:29.358743 1532 log.cc:1196] T 3d540a09c04f459cb2247751a7e6df57 P 0bdc0ce86e3f46c6b6e579d883beabec: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-0/wal/wals/3d540a09c04f459cb2247751a7e6df57
I20260806 10:53:29.359297 1532 ts_tablet_manager.cc:1939] T 3d540a09c04f459cb2247751a7e6df57 P 0bdc0ce86e3f46c6b6e579d883beabec: Deleting consensus metadata
I20260806 10:53:29.359800 1531 ts_tablet_manager.cc:1918] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 10:53:29.359952 1531 log.cc:1196] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/3d540a09c04f459cb2247751a7e6df57
I20260806 10:53:29.360437 1531 ts_tablet_manager.cc:1939] T 3d540a09c04f459cb2247751a7e6df57 P aca3d6c624254657a57a0d6af3e2100d: Deleting consensus metadata
I20260806 10:53:29.360954 897 catalog_manager.cc:4841] TS 0bdc0ce86e3f46c6b6e579d883beabec (127.30.218.1:34647): tablet 3d540a09c04f459cb2247751a7e6df57 (table test-table [id=60a5ae58cdd7402aacfef62bb54c3335]) successfully deleted
I20260806 10:53:29.361635 899 catalog_manager.cc:4841] TS aca3d6c624254657a57a0d6af3e2100d (127.30.218.3:34833): tablet 3d540a09c04f459cb2247751a7e6df57 (table test-table [id=60a5ae58cdd7402aacfef62bb54c3335]) successfully deleted
I20260806 10:53:29.366987 1533 ts_tablet_manager.cc:1918] T 3d540a09c04f459cb2247751a7e6df57 P 7ac2861735ad4be9b977108ebae1058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 10:53:29.367156 1533 log.cc:1196] T 3d540a09c04f459cb2247751a7e6df57 P 7ac2861735ad4be9b977108ebae1058d: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/3d540a09c04f459cb2247751a7e6df57
I20260806 10:53:29.367616 1533 ts_tablet_manager.cc:1939] T 3d540a09c04f459cb2247751a7e6df57 P 7ac2861735ad4be9b977108ebae1058d: Deleting consensus metadata
I20260806 10:53:29.368992 899 catalog_manager.cc:4841] TS 7ac2861735ad4be9b977108ebae1058d (127.30.218.2:40095): tablet 3d540a09c04f459cb2247751a7e6df57 (table test-table [id=60a5ae58cdd7402aacfef62bb54c3335]) successfully deleted
Aug 06 10:53:29 dist-test-slave-49h3 krb5kdc[866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013609, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 10:53:29 dist-test-slave-49h3 krb5kdc[866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013609, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:29.413034 1541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:29.398004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41198 (local address 127.30.218.62:42163)
0806 10:53:29.398638 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:29.398643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:29.398670 (+ 27us) server_negotiation.cc:407] Connection header received
0806 10:53:29.398853 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:29.398858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:29.398934 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:29.399044 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:29.400504 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:29.401410 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:29.402742 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:29.403160 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:29.403267 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:29.407647 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:29.407653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:29.407657 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:29.407698 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:29.410633 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:29.411278 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:29.411282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:29.411285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:29.411406 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:29.411721 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:29.411724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:29.411726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:29.412057 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:29.412264 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:29.412524 (+ 260us) server_negotiation.cc:299] Negotiation successful
0806 10:53:29.412717 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":341,"threads_started":1}
I20260806 10:53:29.416929 911 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:53:29.417708 911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:53:29.425741 1054 tablet_service.cc:1467] Processing CreateTablet for tablet 4170c4197d1e4f8d83e206a077f28043 (DEFAULT_TABLE table=test-table [id=4b0b3df7a3774c51b3b538c690a58665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:29.425886 1197 tablet_service.cc:1467] Processing CreateTablet for tablet 4170c4197d1e4f8d83e206a077f28043 (DEFAULT_TABLE table=test-table [id=4b0b3df7a3774c51b3b538c690a58665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:29.426350 1054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4170c4197d1e4f8d83e206a077f28043. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:29.426403 1197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4170c4197d1e4f8d83e206a077f28043. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:29.430497 1343 tablet_service.cc:1467] Processing CreateTablet for tablet 4170c4197d1e4f8d83e206a077f28043 (DEFAULT_TABLE table=test-table [id=4b0b3df7a3774c51b3b538c690a58665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:29.431011 1343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4170c4197d1e4f8d83e206a077f28043. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:29.432778 1542 tablet_bootstrap.cc:492] T 4170c4197d1e4f8d83e206a077f28043 P 7ac2861735ad4be9b977108ebae1058d: Bootstrap starting.
I20260806 10:53:29.434196 1543 tablet_bootstrap.cc:492] T 4170c4197d1e4f8d83e206a077f28043 P 0bdc0ce86e3f46c6b6e579d883beabec: Bootstrap starting.
I20260806 10:53:29.435377 1542 tablet_bootstrap.cc:654] T 4170c4197d1e4f8d83e206a077f28043 P 7ac2861735ad4be9b977108ebae1058d: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:29.436160 1543 tablet_bootstrap.cc:654] T 4170c4197d1e4f8d83e206a077f28043 P 0bdc0ce86e3f46c6b6e579d883beabec: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:29.438016 1542 tablet_bootstrap.cc:492] T 4170c4197d1e4f8d83e206a077f28043 P 7ac2861735ad4be9b977108ebae1058d: No bootstrap required, opened a new log
I20260806 10:53:29.438549 1543 tablet_bootstrap.cc:492] T 4170c4197d1e4f8d83e206a077f28043 P 0bdc0ce86e3f46c6b6e579d883beabec: No bootstrap required, opened a new log
I20260806 10:53:29.438547 1546 tablet_bootstrap.cc:492] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d: Bootstrap starting.
I20260806 10:53:29.438704 1543 ts_tablet_manager.cc:1397] T 4170c4197d1e4f8d83e206a077f28043 P 0bdc0ce86e3f46c6b6e579d883beabec: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 10:53:29.439514 1542 ts_tablet_manager.cc:1397] T 4170c4197d1e4f8d83e206a077f28043 P 7ac2861735ad4be9b977108ebae1058d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 10:53:29.439652 1543 raft_consensus.cc:348] T 4170c4197d1e4f8d83e206a077f28043 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:29.439954 1543 raft_consensus.cc:374] T 4170c4197d1e4f8d83e206a077f28043 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:29.440062 1543 raft_consensus.cc:729] T 4170c4197d1e4f8d83e206a077f28043 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bdc0ce86e3f46c6b6e579d883beabec, State: Initialized, Role: FOLLOWER
I20260806 10:53:29.440106 1546 tablet_bootstrap.cc:654] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:29.440325 1543 consensus_queue.cc:260] T 4170c4197d1e4f8d83e206a077f28043 P 0bdc0ce86e3f46c6b6e579d883beabec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:29.440481 1542 raft_consensus.cc:348] T 4170c4197d1e4f8d83e206a077f28043 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:29.440780 1542 raft_consensus.cc:374] T 4170c4197d1e4f8d83e206a077f28043 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:29.440891 1542 raft_consensus.cc:729] T 4170c4197d1e4f8d83e206a077f28043 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac2861735ad4be9b977108ebae1058d, State: Initialized, Role: FOLLOWER
I20260806 10:53:29.441155 1542 consensus_queue.cc:260] T 4170c4197d1e4f8d83e206a077f28043 P 7ac2861735ad4be9b977108ebae1058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:29.441401 1543 ts_tablet_manager.cc:1428] T 4170c4197d1e4f8d83e206a077f28043 P 0bdc0ce86e3f46c6b6e579d883beabec: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 10:53:29.442601 1542 ts_tablet_manager.cc:1428] T 4170c4197d1e4f8d83e206a077f28043 P 7ac2861735ad4be9b977108ebae1058d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 10:53:29.443490 1546 tablet_bootstrap.cc:492] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d: No bootstrap required, opened a new log
I20260806 10:53:29.443704 1546 ts_tablet_manager.cc:1397] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260806 10:53:29.444542 1546 raft_consensus.cc:348] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:29.444864 1546 raft_consensus.cc:374] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:29.445006 1546 raft_consensus.cc:729] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aca3d6c624254657a57a0d6af3e2100d, State: Initialized, Role: FOLLOWER
I20260806 10:53:29.445258 1546 consensus_queue.cc:260] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:29.445842 1546 ts_tablet_manager.cc:1428] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 10:53:29.798731 1505 raft_consensus.cc:482] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:53:29.799008 1505 raft_consensus.cc:504] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:29.799887 1505 leader_election.cc:288] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bdc0ce86e3f46c6b6e579d883beabec (127.30.218.1:34647), 7ac2861735ad4be9b977108ebae1058d (127.30.218.2:40095)
I20260806 10:53:29.800666 1076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4170c4197d1e4f8d83e206a077f28043" candidate_uuid: "aca3d6c624254657a57a0d6af3e2100d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" is_pre_election: true
I20260806 10:53:29.800665 1222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4170c4197d1e4f8d83e206a077f28043" candidate_uuid: "aca3d6c624254657a57a0d6af3e2100d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac2861735ad4be9b977108ebae1058d" is_pre_election: true
I20260806 10:53:29.800998 1222 raft_consensus.cc:2454] T 4170c4197d1e4f8d83e206a077f28043 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aca3d6c624254657a57a0d6af3e2100d in term 0.
I20260806 10:53:29.800998 1076 raft_consensus.cc:2454] T 4170c4197d1e4f8d83e206a077f28043 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aca3d6c624254657a57a0d6af3e2100d in term 0.
I20260806 10:53:29.801654 1292 leader_election.cc:302] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0bdc0ce86e3f46c6b6e579d883beabec, 7ac2861735ad4be9b977108ebae1058d, aca3d6c624254657a57a0d6af3e2100d; no voters:
I20260806 10:53:29.801990 1505 raft_consensus.cc:2781] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:53:29.802130 1505 raft_consensus.cc:482] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:53:29.802227 1505 raft_consensus.cc:3037] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:29.805130 1505 raft_consensus.cc:504] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:29.805806 1505 leader_election.cc:288] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d [CANDIDATE]: Term 1 election: Requested vote from peers 0bdc0ce86e3f46c6b6e579d883beabec (127.30.218.1:34647), 7ac2861735ad4be9b977108ebae1058d (127.30.218.2:40095)
I20260806 10:53:29.806461 1076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4170c4197d1e4f8d83e206a077f28043" candidate_uuid: "aca3d6c624254657a57a0d6af3e2100d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bdc0ce86e3f46c6b6e579d883beabec"
I20260806 10:53:29.806738 1076 raft_consensus.cc:3037] T 4170c4197d1e4f8d83e206a077f28043 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:29.810019 1076 raft_consensus.cc:2454] T 4170c4197d1e4f8d83e206a077f28043 P 0bdc0ce86e3f46c6b6e579d883beabec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aca3d6c624254657a57a0d6af3e2100d in term 1.
I20260806 10:53:29.810442 1222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4170c4197d1e4f8d83e206a077f28043" candidate_uuid: "aca3d6c624254657a57a0d6af3e2100d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac2861735ad4be9b977108ebae1058d"
I20260806 10:53:29.810631 1292 leader_election.cc:302] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bdc0ce86e3f46c6b6e579d883beabec, aca3d6c624254657a57a0d6af3e2100d; no voters:
I20260806 10:53:29.810734 1222 raft_consensus.cc:3037] T 4170c4197d1e4f8d83e206a077f28043 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:29.810994 1505 raft_consensus.cc:2781] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:29.811161 1505 raft_consensus.cc:686] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d [term 1 LEADER]: Becoming Leader. State: Replica: aca3d6c624254657a57a0d6af3e2100d, State: Running, Role: LEADER
I20260806 10:53:29.811493 1505 consensus_queue.cc:237] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:29.814026 1222 raft_consensus.cc:2454] T 4170c4197d1e4f8d83e206a077f28043 P 7ac2861735ad4be9b977108ebae1058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aca3d6c624254657a57a0d6af3e2100d in term 1.
I20260806 10:53:29.816412 913 catalog_manager.cc:5495] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d reported cstate change: term changed from 0 to 1, leader changed from <none> to aca3d6c624254657a57a0d6af3e2100d (127.30.218.3). New cstate: current_term: 1 leader_uuid: "aca3d6c624254657a57a0d6af3e2100d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } health_report { overall_health: HEALTHY } } }
I20260806 10:53:29.905884 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:29.900376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35484 (local address 127.30.218.3:34833)
0806 10:53:29.900867 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:29.900872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:29.900891 (+ 19us) server_negotiation.cc:407] Connection header received
0806 10:53:29.900956 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:29.900960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:29.901029 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:29.901115 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:29.901690 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:29.902479 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:29.903860 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:29.904328 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:29.904479 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:29.905039 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:29.905164 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:29.905246 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:29.905419 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 10:53:29.905542 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":266,"threads_started":1}
I20260806 10:53:29.915978 1222 raft_consensus.cc:1261] T 4170c4197d1e4f8d83e206a077f28043 P 7ac2861735ad4be9b977108ebae1058d [term 1 FOLLOWER]: Refusing update from remote peer aca3d6c624254657a57a0d6af3e2100d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:53:29.916016 1076 raft_consensus.cc:1261] T 4170c4197d1e4f8d83e206a077f28043 P 0bdc0ce86e3f46c6b6e579d883beabec [term 1 FOLLOWER]: Refusing update from remote peer aca3d6c624254657a57a0d6af3e2100d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:53:29.916576 1519 consensus_queue.cc:1035] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:53:29.916872 1505 consensus_queue.cc:1035] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:53:29.924058 1552 mvcc.cc:205] Tried to move back new op lower bound from 7315511746211246080 to 7315511745790844928. Current Snapshot: MvccSnapshot[applied={T|T < 7315511746211246080}]
I20260806 10:53:29.976630 913 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:53:29.976931 913 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:53:29.980165 913 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736: Sending DeleteTablet for 3 replicas of tablet 4170c4197d1e4f8d83e206a077f28043
I20260806 10:53:29.981117 1054 tablet_service.cc:1514] Processing DeleteTablet for tablet 4170c4197d1e4f8d83e206a077f28043 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:53:29 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:58990
I20260806 10:53:29.981221 1197 tablet_service.cc:1514] Processing DeleteTablet for tablet 4170c4197d1e4f8d83e206a077f28043 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:53:29 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:37334
I20260806 10:53:29.981266 1343 tablet_service.cc:1514] Processing DeleteTablet for tablet 4170c4197d1e4f8d83e206a077f28043 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:53:29 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:35460
I20260806 10:53:29.982687 911 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:53:29.983579 911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:53:29.988965 1565 tablet_replica.cc:330] T 4170c4197d1e4f8d83e206a077f28043 P 0bdc0ce86e3f46c6b6e579d883beabec: stopping tablet replica
I20260806 10:53:29.989102 1566 tablet_replica.cc:330] T 4170c4197d1e4f8d83e206a077f28043 P 7ac2861735ad4be9b977108ebae1058d: stopping tablet replica
I20260806 10:53:29.989228 1565 raft_consensus.cc:2229] T 4170c4197d1e4f8d83e206a077f28043 P 0bdc0ce86e3f46c6b6e579d883beabec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:53:29.989347 1567 tablet_replica.cc:330] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d: stopping tablet replica
I20260806 10:53:29.989353 1566 raft_consensus.cc:2229] T 4170c4197d1e4f8d83e206a077f28043 P 7ac2861735ad4be9b977108ebae1058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:53:29.989526 1565 raft_consensus.cc:2258] T 4170c4197d1e4f8d83e206a077f28043 P 0bdc0ce86e3f46c6b6e579d883beabec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:53:29.989576 1567 raft_consensus.cc:2229] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:53:29.989611 1566 raft_consensus.cc:2258] T 4170c4197d1e4f8d83e206a077f28043 P 7ac2861735ad4be9b977108ebae1058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:53:29.989989 1567 raft_consensus.cc:2258] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:53:29.990799 1565 ts_tablet_manager.cc:1905] T 4170c4197d1e4f8d83e206a077f28043 P 0bdc0ce86e3f46c6b6e579d883beabec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:53:29.991194 1567 ts_tablet_manager.cc:1905] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:53:29.991420 1566 ts_tablet_manager.cc:1905] T 4170c4197d1e4f8d83e206a077f28043 P 7ac2861735ad4be9b977108ebae1058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:53:29.994202 1054 tablet_service.cc:1467] Processing CreateTablet for tablet 097b3bcfaf9641188e8b8e5ef85bba51 (DEFAULT_TABLE table=test-table [id=236bd3695e6541229e16c289ac1ae570]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:29.994701 1054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097b3bcfaf9641188e8b8e5ef85bba51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:29.995582 1197 tablet_service.cc:1467] Processing CreateTablet for tablet 097b3bcfaf9641188e8b8e5ef85bba51 (DEFAULT_TABLE table=test-table [id=236bd3695e6541229e16c289ac1ae570]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:29.996085 1197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097b3bcfaf9641188e8b8e5ef85bba51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:29.998651 1567 ts_tablet_manager.cc:1918] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:53:29.998768 1566 ts_tablet_manager.cc:1918] T 4170c4197d1e4f8d83e206a077f28043 P 7ac2861735ad4be9b977108ebae1058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:53:29.998914 1566 log.cc:1196] T 4170c4197d1e4f8d83e206a077f28043 P 7ac2861735ad4be9b977108ebae1058d: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/4170c4197d1e4f8d83e206a077f28043
I20260806 10:53:29.998945 1567 log.cc:1196] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/4170c4197d1e4f8d83e206a077f28043
I20260806 10:53:29.999398 1566 ts_tablet_manager.cc:1939] T 4170c4197d1e4f8d83e206a077f28043 P 7ac2861735ad4be9b977108ebae1058d: Deleting consensus metadata
I20260806 10:53:29.999598 1567 ts_tablet_manager.cc:1939] T 4170c4197d1e4f8d83e206a077f28043 P aca3d6c624254657a57a0d6af3e2100d: Deleting consensus metadata
I20260806 10:53:30.000860 899 catalog_manager.cc:4841] TS aca3d6c624254657a57a0d6af3e2100d (127.30.218.3:34833): tablet 4170c4197d1e4f8d83e206a077f28043 (table test-table [id=4b0b3df7a3774c51b3b538c690a58665]) successfully deleted
I20260806 10:53:30.001153 899 catalog_manager.cc:4841] TS 7ac2861735ad4be9b977108ebae1058d (127.30.218.2:40095): tablet 4170c4197d1e4f8d83e206a077f28043 (table test-table [id=4b0b3df7a3774c51b3b538c690a58665]) successfully deleted
I20260806 10:53:30.001322 1343 tablet_service.cc:1467] Processing CreateTablet for tablet 097b3bcfaf9641188e8b8e5ef85bba51 (DEFAULT_TABLE table=test-table [id=236bd3695e6541229e16c289ac1ae570]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:30.001849 1343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097b3bcfaf9641188e8b8e5ef85bba51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:30.003520 1569 tablet_bootstrap.cc:492] T 097b3bcfaf9641188e8b8e5ef85bba51 P 7ac2861735ad4be9b977108ebae1058d: Bootstrap starting.
I20260806 10:53:30.004308 1568 tablet_bootstrap.cc:492] T 097b3bcfaf9641188e8b8e5ef85bba51 P 0bdc0ce86e3f46c6b6e579d883beabec: Bootstrap starting.
I20260806 10:53:30.006138 1568 tablet_bootstrap.cc:654] T 097b3bcfaf9641188e8b8e5ef85bba51 P 0bdc0ce86e3f46c6b6e579d883beabec: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:30.006181 1569 tablet_bootstrap.cc:654] T 097b3bcfaf9641188e8b8e5ef85bba51 P 7ac2861735ad4be9b977108ebae1058d: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:30.008775 1572 tablet_bootstrap.cc:492] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d: Bootstrap starting.
I20260806 10:53:30.012360 1565 ts_tablet_manager.cc:1918] T 4170c4197d1e4f8d83e206a077f28043 P 0bdc0ce86e3f46c6b6e579d883beabec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:53:30.012562 1565 log.cc:1196] T 4170c4197d1e4f8d83e206a077f28043 P 0bdc0ce86e3f46c6b6e579d883beabec: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-0/wal/wals/4170c4197d1e4f8d83e206a077f28043
I20260806 10:53:30.012877 1572 tablet_bootstrap.cc:654] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:30.013123 1565 ts_tablet_manager.cc:1939] T 4170c4197d1e4f8d83e206a077f28043 P 0bdc0ce86e3f46c6b6e579d883beabec: Deleting consensus metadata
I20260806 10:53:30.014816 1568 tablet_bootstrap.cc:492] T 097b3bcfaf9641188e8b8e5ef85bba51 P 0bdc0ce86e3f46c6b6e579d883beabec: No bootstrap required, opened a new log
I20260806 10:53:30.014825 1569 tablet_bootstrap.cc:492] T 097b3bcfaf9641188e8b8e5ef85bba51 P 7ac2861735ad4be9b977108ebae1058d: No bootstrap required, opened a new log
I20260806 10:53:30.014874 897 catalog_manager.cc:4841] TS 0bdc0ce86e3f46c6b6e579d883beabec (127.30.218.1:34647): tablet 4170c4197d1e4f8d83e206a077f28043 (table test-table [id=4b0b3df7a3774c51b3b538c690a58665]) successfully deleted
I20260806 10:53:30.014984 1568 ts_tablet_manager.cc:1397] T 097b3bcfaf9641188e8b8e5ef85bba51 P 0bdc0ce86e3f46c6b6e579d883beabec: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260806 10:53:30.014995 1569 ts_tablet_manager.cc:1397] T 097b3bcfaf9641188e8b8e5ef85bba51 P 7ac2861735ad4be9b977108ebae1058d: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260806 10:53:30.015798 1568 raft_consensus.cc:348] T 097b3bcfaf9641188e8b8e5ef85bba51 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:30.015826 1569 raft_consensus.cc:348] T 097b3bcfaf9641188e8b8e5ef85bba51 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:30.016095 1569 raft_consensus.cc:374] T 097b3bcfaf9641188e8b8e5ef85bba51 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:30.016093 1568 raft_consensus.cc:374] T 097b3bcfaf9641188e8b8e5ef85bba51 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:30.016191 1568 raft_consensus.cc:729] T 097b3bcfaf9641188e8b8e5ef85bba51 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bdc0ce86e3f46c6b6e579d883beabec, State: Initialized, Role: FOLLOWER
I20260806 10:53:30.016196 1569 raft_consensus.cc:729] T 097b3bcfaf9641188e8b8e5ef85bba51 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac2861735ad4be9b977108ebae1058d, State: Initialized, Role: FOLLOWER
I20260806 10:53:30.016470 1569 consensus_queue.cc:260] T 097b3bcfaf9641188e8b8e5ef85bba51 P 7ac2861735ad4be9b977108ebae1058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:30.016487 1568 consensus_queue.cc:260] T 097b3bcfaf9641188e8b8e5ef85bba51 P 0bdc0ce86e3f46c6b6e579d883beabec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:30.017472 1572 tablet_bootstrap.cc:492] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d: No bootstrap required, opened a new log
I20260806 10:53:30.017666 1572 ts_tablet_manager.cc:1397] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.001s
I20260806 10:53:30.017732 1569 ts_tablet_manager.cc:1428] T 097b3bcfaf9641188e8b8e5ef85bba51 P 7ac2861735ad4be9b977108ebae1058d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260806 10:53:30.018025 1568 ts_tablet_manager.cc:1428] T 097b3bcfaf9641188e8b8e5ef85bba51 P 0bdc0ce86e3f46c6b6e579d883beabec: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260806 10:53:30.018462 1572 raft_consensus.cc:348] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:30.018814 1572 raft_consensus.cc:374] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:30.018939 1572 raft_consensus.cc:729] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aca3d6c624254657a57a0d6af3e2100d, State: Initialized, Role: FOLLOWER
I20260806 10:53:30.019266 1572 consensus_queue.cc:260] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:30.019874 1572 ts_tablet_manager.cc:1428] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260806 10:53:30.058624 1555 raft_consensus.cc:482] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:53:30.058872 1555 raft_consensus.cc:504] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:30.059728 1555 leader_election.cc:288] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bdc0ce86e3f46c6b6e579d883beabec (127.30.218.1:34647), 7ac2861735ad4be9b977108ebae1058d (127.30.218.2:40095)
I20260806 10:53:30.060271 1076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097b3bcfaf9641188e8b8e5ef85bba51" candidate_uuid: "aca3d6c624254657a57a0d6af3e2100d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" is_pre_election: true
I20260806 10:53:30.060348 1222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097b3bcfaf9641188e8b8e5ef85bba51" candidate_uuid: "aca3d6c624254657a57a0d6af3e2100d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac2861735ad4be9b977108ebae1058d" is_pre_election: true
I20260806 10:53:30.060557 1076 raft_consensus.cc:2454] T 097b3bcfaf9641188e8b8e5ef85bba51 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aca3d6c624254657a57a0d6af3e2100d in term 0.
I20260806 10:53:30.060583 1222 raft_consensus.cc:2454] T 097b3bcfaf9641188e8b8e5ef85bba51 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aca3d6c624254657a57a0d6af3e2100d in term 0.
I20260806 10:53:30.060983 1292 leader_election.cc:302] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bdc0ce86e3f46c6b6e579d883beabec, aca3d6c624254657a57a0d6af3e2100d; no voters:
I20260806 10:53:30.061306 1555 raft_consensus.cc:2781] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:53:30.061439 1555 raft_consensus.cc:482] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:53:30.061563 1555 raft_consensus.cc:3037] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:30.063656 1555 raft_consensus.cc:504] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:30.064260 1555 leader_election.cc:288] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d [CANDIDATE]: Term 1 election: Requested vote from peers 0bdc0ce86e3f46c6b6e579d883beabec (127.30.218.1:34647), 7ac2861735ad4be9b977108ebae1058d (127.30.218.2:40095)
I20260806 10:53:30.064810 1222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097b3bcfaf9641188e8b8e5ef85bba51" candidate_uuid: "aca3d6c624254657a57a0d6af3e2100d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac2861735ad4be9b977108ebae1058d"
I20260806 10:53:30.064819 1076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097b3bcfaf9641188e8b8e5ef85bba51" candidate_uuid: "aca3d6c624254657a57a0d6af3e2100d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bdc0ce86e3f46c6b6e579d883beabec"
I20260806 10:53:30.065037 1222 raft_consensus.cc:3037] T 097b3bcfaf9641188e8b8e5ef85bba51 P 7ac2861735ad4be9b977108ebae1058d [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:30.065047 1076 raft_consensus.cc:3037] T 097b3bcfaf9641188e8b8e5ef85bba51 P 0bdc0ce86e3f46c6b6e579d883beabec [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:30.067390 1076 raft_consensus.cc:2454] T 097b3bcfaf9641188e8b8e5ef85bba51 P 0bdc0ce86e3f46c6b6e579d883beabec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aca3d6c624254657a57a0d6af3e2100d in term 1.
I20260806 10:53:30.067407 1222 raft_consensus.cc:2454] T 097b3bcfaf9641188e8b8e5ef85bba51 P 7ac2861735ad4be9b977108ebae1058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aca3d6c624254657a57a0d6af3e2100d in term 1.
I20260806 10:53:30.067909 1294 leader_election.cc:302] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ac2861735ad4be9b977108ebae1058d, aca3d6c624254657a57a0d6af3e2100d; no voters:
I20260806 10:53:30.068265 1555 raft_consensus.cc:2781] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:30.068416 1555 raft_consensus.cc:686] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d [term 1 LEADER]: Becoming Leader. State: Replica: aca3d6c624254657a57a0d6af3e2100d, State: Running, Role: LEADER
I20260806 10:53:30.068704 1555 consensus_queue.cc:237] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } }
I20260806 10:53:30.071523 911 catalog_manager.cc:5495] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d reported cstate change: term changed from 0 to 1, leader changed from <none> to aca3d6c624254657a57a0d6af3e2100d (127.30.218.3). New cstate: current_term: 1 leader_uuid: "aca3d6c624254657a57a0d6af3e2100d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aca3d6c624254657a57a0d6af3e2100d" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 34833 } health_report { overall_health: HEALTHY } } }
I20260806 10:53:30.109671 1076 raft_consensus.cc:1261] T 097b3bcfaf9641188e8b8e5ef85bba51 P 0bdc0ce86e3f46c6b6e579d883beabec [term 1 FOLLOWER]: Refusing update from remote peer aca3d6c624254657a57a0d6af3e2100d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:53:30.109704 1222 raft_consensus.cc:1261] T 097b3bcfaf9641188e8b8e5ef85bba51 P 7ac2861735ad4be9b977108ebae1058d [term 1 FOLLOWER]: Refusing update from remote peer aca3d6c624254657a57a0d6af3e2100d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:53:30.110240 1505 consensus_queue.cc:1035] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bdc0ce86e3f46c6b6e579d883beabec" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:53:30.110533 1519 consensus_queue.cc:1035] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac2861735ad4be9b977108ebae1058d" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 10:53:30.130487 1551 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 10:53:31.721597 913 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:53:31.721907 913 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:53:31.726831 913 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 085d238d8bf440d685ad93529dc3e736: Sending DeleteTablet for 3 replicas of tablet 097b3bcfaf9641188e8b8e5ef85bba51
I20260806 10:53:31.727749 1054 tablet_service.cc:1514] Processing DeleteTablet for tablet 097b3bcfaf9641188e8b8e5ef85bba51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:53:31 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:58990
I20260806 10:53:31.727843 1197 tablet_service.cc:1514] Processing DeleteTablet for tablet 097b3bcfaf9641188e8b8e5ef85bba51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:53:31 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:37334
I20260806 10:53:31.727833 1343 tablet_service.cc:1514] Processing DeleteTablet for tablet 097b3bcfaf9641188e8b8e5ef85bba51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:53:31 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:35460
I20260806 10:53:31.728554 1598 tablet_replica.cc:330] T 097b3bcfaf9641188e8b8e5ef85bba51 P 0bdc0ce86e3f46c6b6e579d883beabec: stopping tablet replica
I20260806 10:53:31.728837 1600 tablet_replica.cc:330] T 097b3bcfaf9641188e8b8e5ef85bba51 P 7ac2861735ad4be9b977108ebae1058d: stopping tablet replica
I20260806 10:53:31.728857 1598 raft_consensus.cc:2229] T 097b3bcfaf9641188e8b8e5ef85bba51 P 0bdc0ce86e3f46c6b6e579d883beabec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:53:31.729048 1600 raft_consensus.cc:2229] T 097b3bcfaf9641188e8b8e5ef85bba51 P 7ac2861735ad4be9b977108ebae1058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:53:31.729153 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 988
I20260806 10:53:31.729331 1600 raft_consensus.cc:2258] T 097b3bcfaf9641188e8b8e5ef85bba51 P 7ac2861735ad4be9b977108ebae1058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:53:31.729343 1598 raft_consensus.cc:2258] T 097b3bcfaf9641188e8b8e5ef85bba51 P 0bdc0ce86e3f46c6b6e579d883beabec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:53:31.730427 1600 ts_tablet_manager.cc:1905] T 097b3bcfaf9641188e8b8e5ef85bba51 P 7ac2861735ad4be9b977108ebae1058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:53:31.731983 1598 ts_tablet_manager.cc:1905] T 097b3bcfaf9641188e8b8e5ef85bba51 P 0bdc0ce86e3f46c6b6e579d883beabec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:53:31.736132 1599 tablet_replica.cc:330] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d: stopping tablet replica
I20260806 10:53:31.736388 1599 raft_consensus.cc:2229] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:53:31.736913 1599 raft_consensus.cc:2258] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:53:31.738011 1599 ts_tablet_manager.cc:1905] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:53:31.738991 1600 ts_tablet_manager.cc:1918] T 097b3bcfaf9641188e8b8e5ef85bba51 P 7ac2861735ad4be9b977108ebae1058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 10:53:31.739204 1600 log.cc:1196] T 097b3bcfaf9641188e8b8e5ef85bba51 P 7ac2861735ad4be9b977108ebae1058d: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/097b3bcfaf9641188e8b8e5ef85bba51
I20260806 10:53:31.739744 1600 ts_tablet_manager.cc:1939] T 097b3bcfaf9641188e8b8e5ef85bba51 P 7ac2861735ad4be9b977108ebae1058d: Deleting consensus metadata
I20260806 10:53:31.740257 1598 ts_tablet_manager.cc:1918] T 097b3bcfaf9641188e8b8e5ef85bba51 P 0bdc0ce86e3f46c6b6e579d883beabec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 10:53:31.740520 1598 log.cc:1196] T 097b3bcfaf9641188e8b8e5ef85bba51 P 0bdc0ce86e3f46c6b6e579d883beabec: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-0/wal/wals/097b3bcfaf9641188e8b8e5ef85bba51
I20260806 10:53:31.741067 1598 ts_tablet_manager.cc:1939] T 097b3bcfaf9641188e8b8e5ef85bba51 P 0bdc0ce86e3f46c6b6e579d883beabec: Deleting consensus metadata
I20260806 10:53:31.741283 899 catalog_manager.cc:4841] TS 7ac2861735ad4be9b977108ebae1058d (127.30.218.2:40095): tablet 097b3bcfaf9641188e8b8e5ef85bba51 (table test-table [id=236bd3695e6541229e16c289ac1ae570]) successfully deleted
I20260806 10:53:31.742311 897 catalog_manager.cc:4841] TS 0bdc0ce86e3f46c6b6e579d883beabec (127.30.218.1:34647): tablet 097b3bcfaf9641188e8b8e5ef85bba51 (table test-table [id=236bd3695e6541229e16c289ac1ae570]) successfully deleted
I20260806 10:53:31.744974 1599 ts_tablet_manager.cc:1918] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 10:53:31.745127 1599 log.cc:1196] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/097b3bcfaf9641188e8b8e5ef85bba51
I20260806 10:53:31.745604 1599 ts_tablet_manager.cc:1939] T 097b3bcfaf9641188e8b8e5ef85bba51 P aca3d6c624254657a57a0d6af3e2100d: Deleting consensus metadata
I20260806 10:53:31.746697 899 catalog_manager.cc:4841] TS aca3d6c624254657a57a0d6af3e2100d (127.30.218.3:34833): tablet 097b3bcfaf9641188e8b8e5ef85bba51 (table test-table [id=236bd3695e6541229e16c289ac1ae570]) successfully deleted
Aug 06 10:53:31 dist-test-slave-49h3 krb5kdc[866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013609, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.218.1@KRBTEST.COM
I20260806 10:53:31.758865 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:31.732413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58996 (local address 127.30.218.1:34647)
0806 10:53:31.736409 (+ 3996us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:31.736417 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:31.737095 (+ 678us) server_negotiation.cc:407] Connection header received
0806 10:53:31.737252 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:31.737258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:31.737350 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:31.737487 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:31.739460 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:31.740708 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:31.743029 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:31.743814 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:31.743942 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:31.752717 (+ 8775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:31.752738 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:31.752762 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:31.752814 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:31.756013 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:31.756721 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:31.756731 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:31.756733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:31.756892 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:31.757225 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:31.757228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:31.757231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:31.757691 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:31.757966 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:31.758312 (+ 346us) server_negotiation.cc:299] Negotiation successful
0806 10:53:31.758541 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3811,"thread_start_us":3661,"threads_started":1}
W20260806 10:53:31.759688 1122 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58996
W20260806 10:53:31.760478 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 988: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:53:31.760737 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 988
I20260806 10:53:31.783772 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 1131
Aug 06 10:53:31 dist-test-slave-49h3 krb5kdc[866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013609, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.218.2@KRBTEST.COM
I20260806 10:53:31.800138 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:31.784674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37360 (local address 127.30.218.2:40095)
0806 10:53:31.785282 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:31.785294 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:31.785311 (+ 17us) server_negotiation.cc:407] Connection header received
0806 10:53:31.785379 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:31.785384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:31.785487 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:31.785599 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:31.787119 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:31.787947 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:31.789297 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:31.789744 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:31.789863 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:31.794233 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:31.794247 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:31.794265 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:31.794311 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:31.797216 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:31.797946 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:31.797956 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:31.797958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:31.798109 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:31.798481 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:31.798486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:31.798489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:31.798903 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:31.799213 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:31.799521 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 10:53:31.799770 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":277,"threads_started":1}
W20260806 10:53:31.800879 1266 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37360
W20260806 10:53:31.801581 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 1131: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:53:31.801734 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 1131
I20260806 10:53:31.830911 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 1275
Aug 06 10:53:31 dist-test-slave-49h3 krb5kdc[866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013609, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.218.3@KRBTEST.COM
I20260806 10:53:31.848742 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:31.832020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35492 (local address 127.30.218.3:34833)
0806 10:53:31.832703 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:31.832708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:31.832726 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:53:31.832796 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:31.832801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:31.832873 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:31.832983 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:31.834615 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:31.835539 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:31.837028 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:31.837605 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:31.837725 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:31.842552 (+ 4827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:31.842570 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:31.842593 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:31.842630 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:31.845774 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:31.846489 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:31.846498 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:31.846501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:31.846666 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:31.847039 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:31.847043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:31.847045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:31.847507 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:31.847753 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:31.848135 (+ 382us) server_negotiation.cc:299] Negotiation successful
0806 10:53:31.848331 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":363,"threads_started":1}
W20260806 10:53:31.849540 1413 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35492
W20260806 10:53:31.850219 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 1275: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:53:31.850401 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 1275
I20260806 10:53:31.880928 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 880
I20260806 10:53:31.882432 955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:32.020780 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 880
2026-08-06T10:53:32Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9185 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:53:32 dist-test-slave-49h3 krb5kdc[1609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:53:32 dist-test-slave-49h3 krb5kdc[1609](info): set up 2 sockets
Aug 06 10:53:32 dist-test-slave-49h3 krb5kdc[1609](info): commencing operation
krb5kdc: starting...
W20260806 10:53:33.268162 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.196s user 0.001s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260806 10:53:33.369274 31592 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.101s user 0.000s sys 0.025s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:53:33 dist-test-slave-49h3 krb5kdc[1609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:53:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:53:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:53:33.707294 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:39901
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:39901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:34.404016 1623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:34.404657 1623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:34.404969 1623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:34.419680 1623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:53:34.419920 1623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:34.420065 1623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:34.420276 1623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:53:34.420356 1623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:53:34.438799 1623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:39901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:39901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:34.440258 1623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:34.441704 1623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:34.454624 1629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:34.454658 1628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:34.454586 1631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:34.455557 1623 server_base.cc:1034] running on GCE node
I20260806 10:53:34.456568 1623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:34.458338 1623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:34.459570 1623 hybrid_clock.cc:648] HybridClock initialized: now 1786013614459509 us; error 58 us; skew 500 ppm
I20260806 10:53:34.459931 1623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:34 dist-test-slave-49h3 krb5kdc[1609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013614, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:34.463347 1623 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:53:34.464963 1623 webserver.cc:460] Webserver started at http://127.30.218.62:33443/ using document root <none> and password file <none>
I20260806 10:53:34.465761 1623 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:34.465886 1623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:34.466229 1623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:34.469372 1623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "d158fb3c21734822b6e7feabbba9eb3f"
format_stamp: "Formatted at 2026-08-06 10:53:34 on dist-test-slave-49h3"
server_key: "05dbcb8bfd44ce8c7b06adfc8f5a4b6f"
server_key_iv: "79ee7b2af78069413159a09ccba38b89"
server_key_version: "encryptionkey@0"
I20260806 10:53:34.470214 1623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "d158fb3c21734822b6e7feabbba9eb3f"
format_stamp: "Formatted at 2026-08-06 10:53:34 on dist-test-slave-49h3"
server_key: "05dbcb8bfd44ce8c7b06adfc8f5a4b6f"
server_key_iv: "79ee7b2af78069413159a09ccba38b89"
server_key_version: "encryptionkey@0"
I20260806 10:53:34.476253 1623 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 10:53:34.480206 1638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:34.481866 1623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 10:53:34.482067 1623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "d158fb3c21734822b6e7feabbba9eb3f"
format_stamp: "Formatted at 2026-08-06 10:53:34 on dist-test-slave-49h3"
server_key: "05dbcb8bfd44ce8c7b06adfc8f5a4b6f"
server_key_iv: "79ee7b2af78069413159a09ccba38b89"
server_key_version: "encryptionkey@0"
I20260806 10:53:34.482288 1623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:34.504544 1623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:34.509423 1623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:34.509802 1623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:34.529409 1623 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:39901
I20260806 10:53:34.529428 1692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:39901 every 8 connection(s)
I20260806 10:53:34.530967 1623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:53:34.535075 1693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:34.535578 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 1623
I20260806 10:53:34.535782 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:53:34.536490 31592 external_mini_cluster.cc:1428] Setting key 2ff1e1a1d76ee4a6512c87d6a5706145
I20260806 10:53:34.548678 1693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d158fb3c21734822b6e7feabbba9eb3f: Bootstrap starting.
Aug 06 10:53:34 dist-test-slave-49h3 krb5kdc[1609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:34.552225 1693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d158fb3c21734822b6e7feabbba9eb3f: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:34.553498 1693 log.cc:824] T 00000000000000000000000000000000 P d158fb3c21734822b6e7feabbba9eb3f: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:34.556906 1693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d158fb3c21734822b6e7feabbba9eb3f: No bootstrap required, opened a new log
I20260806 10:53:34.560778 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:34.538786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52904 (local address 127.30.218.62:39901)
0806 10:53:34.539661 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:34.539674 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:34.539722 (+ 48us) server_negotiation.cc:407] Connection header received
0806 10:53:34.541697 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:34.541727 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:34.542226 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:34.542646 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:34.544355 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:34.545884 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:34.547362 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:34.548108 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:34.548298 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:34.552773 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:34.552783 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:34.552802 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:34.552854 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:34.556134 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:34.556953 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:34.556963 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:34.556973 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:34.557190 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:34.557651 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:34.557656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:34.557660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:34.558265 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:34.558580 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:34.559139 (+ 559us) server_negotiation.cc:299] Negotiation successful
0806 10:53:34.559616 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":649,"thread_start_us":382,"threads_started":1}
I20260806 10:53:34.565544 1693 raft_consensus.cc:348] T 00000000000000000000000000000000 P d158fb3c21734822b6e7feabbba9eb3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d158fb3c21734822b6e7feabbba9eb3f" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 39901 } }
I20260806 10:53:34.565979 1693 raft_consensus.cc:374] T 00000000000000000000000000000000 P d158fb3c21734822b6e7feabbba9eb3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:34.566105 1693 raft_consensus.cc:729] T 00000000000000000000000000000000 P d158fb3c21734822b6e7feabbba9eb3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d158fb3c21734822b6e7feabbba9eb3f, State: Initialized, Role: FOLLOWER
I20260806 10:53:34.566764 1693 consensus_queue.cc:260] T 00000000000000000000000000000000 P d158fb3c21734822b6e7feabbba9eb3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d158fb3c21734822b6e7feabbba9eb3f" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 39901 } }
I20260806 10:53:34.567015 1693 raft_consensus.cc:388] T 00000000000000000000000000000000 P d158fb3c21734822b6e7feabbba9eb3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:53:34.567153 1693 raft_consensus.cc:482] T 00000000000000000000000000000000 P d158fb3c21734822b6e7feabbba9eb3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:53:34.567320 1693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d158fb3c21734822b6e7feabbba9eb3f [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:34.569800 1693 raft_consensus.cc:504] T 00000000000000000000000000000000 P d158fb3c21734822b6e7feabbba9eb3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d158fb3c21734822b6e7feabbba9eb3f" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 39901 } }
I20260806 10:53:34.570365 1693 leader_election.cc:302] T 00000000000000000000000000000000 P d158fb3c21734822b6e7feabbba9eb3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d158fb3c21734822b6e7feabbba9eb3f; no voters:
I20260806 10:53:34.570982 1693 leader_election.cc:288] T 00000000000000000000000000000000 P d158fb3c21734822b6e7feabbba9eb3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:53:34.571126 1698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d158fb3c21734822b6e7feabbba9eb3f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:34.571875 1698 raft_consensus.cc:686] T 00000000000000000000000000000000 P d158fb3c21734822b6e7feabbba9eb3f [term 1 LEADER]: Becoming Leader. State: Replica: d158fb3c21734822b6e7feabbba9eb3f, State: Running, Role: LEADER
I20260806 10:53:34.572453 1698 consensus_queue.cc:237] T 00000000000000000000000000000000 P d158fb3c21734822b6e7feabbba9eb3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d158fb3c21734822b6e7feabbba9eb3f" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 39901 } }
I20260806 10:53:34.573173 1693 sys_catalog.cc:564] T 00000000000000000000000000000000 P d158fb3c21734822b6e7feabbba9eb3f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:53:34.575457 1700 sys_catalog.cc:455] T 00000000000000000000000000000000 P d158fb3c21734822b6e7feabbba9eb3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d158fb3c21734822b6e7feabbba9eb3f. Latest consensus state: current_term: 1 leader_uuid: "d158fb3c21734822b6e7feabbba9eb3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d158fb3c21734822b6e7feabbba9eb3f" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 39901 } } }
I20260806 10:53:34.575904 1700 sys_catalog.cc:458] T 00000000000000000000000000000000 P d158fb3c21734822b6e7feabbba9eb3f [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:34.577893 1699 sys_catalog.cc:455] T 00000000000000000000000000000000 P d158fb3c21734822b6e7feabbba9eb3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d158fb3c21734822b6e7feabbba9eb3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d158fb3c21734822b6e7feabbba9eb3f" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 39901 } } }
I20260806 10:53:34.578183 1699 sys_catalog.cc:458] T 00000000000000000000000000000000 P d158fb3c21734822b6e7feabbba9eb3f [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:34.578749 1705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:53:34.583964 1705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:53:34.593858 1705 catalog_manager.cc:1347] Generated new cluster ID: 26773697c88e4d379e3c68fbde0187f1
I20260806 10:53:34.594046 1705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:53:34.610136 1705 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:53:34.611651 1705 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:53:34.623172 1705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d158fb3c21734822b6e7feabbba9eb3f: Generated new TSK 0
I20260806 10:53:34.624627 1705 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
I20260806 10:53:34.701270 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:39901
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:35.058457 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:35.058785 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:35.058920 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:35.066763 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:35.066906 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:35.067081 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:53:35.077617 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:39901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:35.079073 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:35.080389 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:35.093982 1726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:35.094017 1727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:35.093962 1729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:35.095054 1720 server_base.cc:1034] running on GCE node
I20260806 10:53:35.095652 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:35.096673 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:35.097906 1720 hybrid_clock.cc:648] HybridClock initialized: now 1786013615097856 us; error 39 us; skew 500 ppm
I20260806 10:53:35.098284 1720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:35 dist-test-slave-49h3 krb5kdc[1609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013615, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:35.101744 1720 init.cc:377] Logged in from keytab as kudu/127.30.218.1@KRBTEST.COM (short username kudu)
I20260806 10:53:35.103299 1720 webserver.cc:460] Webserver started at http://127.30.218.1:43951/ using document root <none> and password file <none>
I20260806 10:53:35.104017 1720 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:35.104151 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:35.104461 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:35.107656 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "4d185b1ac0d64ab29cd637362def9fbb"
format_stamp: "Formatted at 2026-08-06 10:53:35 on dist-test-slave-49h3"
server_key: "8088587e8c1935ed80e6e599b0ae4f33"
server_key_iv: "b4419a70a7277a6ea7b818d0e5746a2c"
server_key_version: "encryptionkey@0"
I20260806 10:53:35.108503 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "4d185b1ac0d64ab29cd637362def9fbb"
format_stamp: "Formatted at 2026-08-06 10:53:35 on dist-test-slave-49h3"
server_key: "8088587e8c1935ed80e6e599b0ae4f33"
server_key_iv: "b4419a70a7277a6ea7b818d0e5746a2c"
server_key_version: "encryptionkey@0"
I20260806 10:53:35.114181 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 10:53:35.118045 1736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:35.119514 1720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:53:35.119693 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "4d185b1ac0d64ab29cd637362def9fbb"
format_stamp: "Formatted at 2026-08-06 10:53:35 on dist-test-slave-49h3"
server_key: "8088587e8c1935ed80e6e599b0ae4f33"
server_key_iv: "b4419a70a7277a6ea7b818d0e5746a2c"
server_key_version: "encryptionkey@0"
I20260806 10:53:35.119907 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:35.149168 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:35.154121 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:35.154451 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:35.155498 1720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:53:35.156978 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:35.157070 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:35.157194 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:35.157246 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:35.189843 1720 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:45029
I20260806 10:53:35.189893 1861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:45029 every 8 connection(s)
I20260806 10:53:35.191217 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:53:35.196926 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 1720
I20260806 10:53:35.197131 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:53:35.197830 31592 external_mini_cluster.cc:1428] Setting key aaa27254a6331fc7aacccfb39a846519
Aug 06 10:53:35 dist-test-slave-49h3 krb5kdc[1609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013615, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:35.213008 1864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:35.193907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:45389 (local address 127.30.218.62:39901)
0806 10:53:35.194423 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:35.194428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:35.195563 (+ 1135us) server_negotiation.cc:407] Connection header received
0806 10:53:35.197563 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:35.197569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:35.197672 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:35.197772 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:35.199590 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:35.200343 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:35.202101 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:35.202522 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:35.202654 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:35.206916 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:35.206921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:35.206925 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:35.206966 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:35.209643 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:35.210491 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:35.210496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:35.210498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:35.210633 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:35.211095 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:35.211106 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:35.211109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:35.211405 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:35.211552 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:35.212360 (+ 808us) server_negotiation.cc:299] Negotiation successful
0806 10:53:35.212631 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":261,"threads_started":1}
I20260806 10:53:35.214094 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:35.194626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39901 (local address 127.30.218.1:45389)
0806 10:53:35.195386 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:35.195426 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:35.197237 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:35.198062 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:35.198072 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:35.198685 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:35.199403 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:35.199425 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:35.200520 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:35.200524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:35.201932 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:35.201948 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:35.202775 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:35.202782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:35.203045 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:35.203773 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:35.203797 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:35.206624 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:35.209832 (+ 3208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:35.209842 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:35.209858 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:35.210345 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:35.210750 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:35.210755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:35.210759 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:35.210969 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:35.211544 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:35.211552 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:35.211949 (+ 397us) client_negotiation.cc:769] Sending connection context
0806 10:53:35.212213 (+ 264us) client_negotiation.cc:240] Negotiation successful
0806 10:53:35.212842 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":289,"threads_started":1}
I20260806 10:53:35.215747 1862 heartbeater.cc:347] Connected to a master server at 127.30.218.62:39901
I20260806 10:53:35.216173 1862 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:35.217178 1862 heartbeater.cc:511] Master 127.30.218.62:39901 requested a full tablet report, sending...
I20260806 10:53:35.219746 1656 ts_manager.cc:194] Registered new tserver with Master: 4d185b1ac0d64ab29cd637362def9fbb (127.30.218.1:45029)
WARNING: no policy specified for kudu/127.30.218.2@KRBTEST.COM; defaulting to no policy
I20260806 10:53:35.221338 1656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.1@KRBTEST.COM'} at 127.30.218.1:45389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
I20260806 10:53:35.249701 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:39901
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:35.613853 1869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:35.614221 1869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:35.614374 1869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:35.622171 1869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:35.622334 1869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:35.622493 1869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:53:35.632982 1869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:39901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:35.634485 1869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:35.635908 1869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:35.650008 1877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:35.650084 1874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:35.650043 1875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:35.659257 1869 server_base.cc:1034] running on GCE node
I20260806 10:53:35.660014 1869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:35.661222 1869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:35.662459 1869 hybrid_clock.cc:648] HybridClock initialized: now 1786013615662400 us; error 61 us; skew 500 ppm
I20260806 10:53:35.662817 1869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:35 dist-test-slave-49h3 krb5kdc[1609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013615, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:35.666321 1869 init.cc:377] Logged in from keytab as kudu/127.30.218.2@KRBTEST.COM (short username kudu)
I20260806 10:53:35.668053 1869 webserver.cc:460] Webserver started at http://127.30.218.2:34209/ using document root <none> and password file <none>
I20260806 10:53:35.668840 1869 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:35.668970 1869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:35.669392 1869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:35.672760 1869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "f97c940fd90444e8ba3a809e39d060b3"
format_stamp: "Formatted at 2026-08-06 10:53:35 on dist-test-slave-49h3"
server_key: "62bf928c02f89c654d91282f34e89604"
server_key_iv: "d62db3c27fe409896a010e3a517a2175"
server_key_version: "encryptionkey@0"
I20260806 10:53:35.673698 1869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "f97c940fd90444e8ba3a809e39d060b3"
format_stamp: "Formatted at 2026-08-06 10:53:35 on dist-test-slave-49h3"
server_key: "62bf928c02f89c654d91282f34e89604"
server_key_iv: "d62db3c27fe409896a010e3a517a2175"
server_key_version: "encryptionkey@0"
I20260806 10:53:35.679430 1869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 10:53:35.683491 1885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:35.685055 1869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:53:35.685251 1869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "f97c940fd90444e8ba3a809e39d060b3"
format_stamp: "Formatted at 2026-08-06 10:53:35 on dist-test-slave-49h3"
server_key: "62bf928c02f89c654d91282f34e89604"
server_key_iv: "d62db3c27fe409896a010e3a517a2175"
server_key_version: "encryptionkey@0"
I20260806 10:53:35.685514 1869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:35.706307 1869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:35.711009 1869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:35.711314 1869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:35.712402 1869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:53:35.713954 1869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:35.714038 1869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:35.714179 1869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:35.714236 1869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:35.747179 1869 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:34611
I20260806 10:53:35.747215 2004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:34611 every 8 connection(s)
I20260806 10:53:35.748610 1869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:53:35.753059 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 1869
I20260806 10:53:35.753300 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:53:35.753851 31592 external_mini_cluster.cc:1428] Setting key 4895b8a628d2b64f67bb02051ec2bc2e
Aug 06 10:53:35 dist-test-slave-49h3 krb5kdc[1609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013615, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:35.770066 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:35.751350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:46803 (local address 127.30.218.62:39901)
0806 10:53:35.751820 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:35.751824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:35.753324 (+ 1500us) server_negotiation.cc:407] Connection header received
0806 10:53:35.755020 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:35.755026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:35.755126 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:35.755237 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:35.756978 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:35.757778 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:35.759436 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:35.759898 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:35.760014 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:35.763739 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:35.763745 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:35.763749 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:35.763787 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:35.766622 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:35.767481 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:35.767486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:35.767488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:35.767630 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:35.768096 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:35.768100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:35.768102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:35.768405 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:35.768547 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:35.769337 (+ 790us) server_negotiation.cc:299] Negotiation successful
0806 10:53:35.769632 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":337,"thread_start_us":232,"threads_started":1}
I20260806 10:53:35.771209 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:35.751838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39901 (local address 127.30.218.2:46803)
0806 10:53:35.752613 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:35.752663 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:35.754690 (+ 2027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:35.755527 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:35.755537 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:35.756173 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:35.756797 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:35.756817 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:35.757936 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:35.757940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:35.759284 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:35.759299 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:35.760141 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:35.760149 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:35.760395 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:35.761095 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:35.761126 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:35.763461 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:35.766805 (+ 3344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:35.766816 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:35.766836 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:35.767349 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:35.767746 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:35.767751 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:35.767756 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:35.767979 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:35.768540 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:35.768549 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:35.768948 (+ 399us) client_negotiation.cc:769] Sending connection context
0806 10:53:35.769222 (+ 274us) client_negotiation.cc:240] Negotiation successful
0806 10:53:35.769893 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":311,"threads_started":1}
I20260806 10:53:35.772873 2005 heartbeater.cc:347] Connected to a master server at 127.30.218.62:39901
I20260806 10:53:35.773188 2005 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:35.774077 2005 heartbeater.cc:511] Master 127.30.218.62:39901 requested a full tablet report, sending...
I20260806 10:53:35.775830 1656 ts_manager.cc:194] Registered new tserver with Master: f97c940fd90444e8ba3a809e39d060b3 (127.30.218.2:34611)
WARNING: no policy specified for kudu/127.30.218.3@KRBTEST.COM; defaulting to no policy
I20260806 10:53:35.776942 1656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.2@KRBTEST.COM'} at 127.30.218.2:46803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
I20260806 10:53:35.805879 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:39901
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:36.154981 2012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:36.155334 2012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:36.155470 2012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:36.163273 2012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:36.163446 2012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:36.163606 2012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:53:36.174376 2012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:39901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:36.175822 2012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:36.177263 2012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:36.192400 2017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:36.192899 2018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:36.193094 2020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:36.193732 2012 server_base.cc:1034] running on GCE node
I20260806 10:53:36.194438 2012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:36.195531 2012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:36.196754 2012 hybrid_clock.cc:648] HybridClock initialized: now 1786013616196684 us; error 60 us; skew 500 ppm
I20260806 10:53:36.197096 2012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:36 dist-test-slave-49h3 krb5kdc[1609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013616, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:36.200462 2012 init.cc:377] Logged in from keytab as kudu/127.30.218.3@KRBTEST.COM (short username kudu)
I20260806 10:53:36.202153 2012 webserver.cc:460] Webserver started at http://127.30.218.3:41343/ using document root <none> and password file <none>
I20260806 10:53:36.202934 2012 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:36.203063 2012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:36.203404 2012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:36.206838 2012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "696281a1ea53466389341b52aa34ab55"
format_stamp: "Formatted at 2026-08-06 10:53:36 on dist-test-slave-49h3"
server_key: "090e9fcd10197ee54edb4b381576764b"
server_key_iv: "44f3f9d7e99a12a96344121ccc11600d"
server_key_version: "encryptionkey@0"
I20260806 10:53:36.207762 2012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "696281a1ea53466389341b52aa34ab55"
format_stamp: "Formatted at 2026-08-06 10:53:36 on dist-test-slave-49h3"
server_key: "090e9fcd10197ee54edb4b381576764b"
server_key_iv: "44f3f9d7e99a12a96344121ccc11600d"
server_key_version: "encryptionkey@0"
I20260806 10:53:36.213806 2012 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 10:53:36.217913 2027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:36.219412 2012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:53:36.219604 2012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "696281a1ea53466389341b52aa34ab55"
format_stamp: "Formatted at 2026-08-06 10:53:36 on dist-test-slave-49h3"
server_key: "090e9fcd10197ee54edb4b381576764b"
server_key_iv: "44f3f9d7e99a12a96344121ccc11600d"
server_key_version: "encryptionkey@0"
I20260806 10:53:36.219830 2012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:36.225256 1862 heartbeater.cc:503] Master 127.30.218.62:39901 was elected leader, sending a full tablet report...
I20260806 10:53:36.240070 2012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:36.244655 2012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:36.245003 2012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:36.246119 2012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:53:36.247663 2012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:36.247747 2012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:36.247849 2012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:36.247895 2012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:36.280892 2012 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:45849
I20260806 10:53:36.280922 2146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:45849 every 8 connection(s)
I20260806 10:53:36.282546 2012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:53:36.291898 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 2012
I20260806 10:53:36.292057 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:53:36.292515 31592 external_mini_cluster.cc:1428] Setting key 2324b5e73a3354cf64f161123f5c5c61
Aug 06 10:53:36 dist-test-slave-49h3 krb5kdc[1609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013616, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:36.304952 2149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:36.285268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:49357 (local address 127.30.218.62:39901)
0806 10:53:36.285836 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:36.285842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:36.287129 (+ 1287us) server_negotiation.cc:407] Connection header received
0806 10:53:36.289102 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:36.289107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:36.289220 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:36.289356 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:36.291205 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:36.292067 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:36.293982 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:36.294601 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:36.294741 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:36.298983 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:36.298989 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:36.298992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:36.299033 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:36.301718 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:36.302578 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:36.302581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:36.302583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:36.302721 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:36.303141 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:36.303144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:36.303146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:36.303447 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:36.303575 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:36.304392 (+ 817us) server_negotiation.cc:299] Negotiation successful
0806 10:53:36.304606 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":261,"threads_started":1}
I20260806 10:53:36.306077 2150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:36.285877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39901 (local address 127.30.218.3:49357)
0806 10:53:36.286936 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:36.286979 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:36.288790 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:36.289622 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:36.289632 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:36.290275 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:36.290982 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:36.291002 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:36.292282 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:36.292286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:36.293830 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:36.293845 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:36.294855 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:36.294864 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:36.295215 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:36.296022 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:36.296057 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:36.298694 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:36.301900 (+ 3206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:36.301909 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:36.301926 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:36.302466 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:36.302802 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:36.302805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:36.302808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:36.303049 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:36.303591 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:36.303598 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:36.303985 (+ 387us) client_negotiation.cc:769] Sending connection context
0806 10:53:36.304267 (+ 282us) client_negotiation.cc:240] Negotiation successful
0806 10:53:36.304725 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":343,"threads_started":1}
I20260806 10:53:36.307811 2147 heartbeater.cc:347] Connected to a master server at 127.30.218.62:39901
I20260806 10:53:36.308162 2147 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:36.309036 2147 heartbeater.cc:511] Master 127.30.218.62:39901 requested a full tablet report, sending...
I20260806 10:53:36.310657 1656 ts_manager.cc:194] Registered new tserver with Master: 696281a1ea53466389341b52aa34ab55 (127.30.218.3:45849)
I20260806 10:53:36.311498 1656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.3@KRBTEST.COM'} at 127.30.218.3:49357
I20260806 10:53:36.321285 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 10:53:36.365613 2149 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:53:36.362868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37518 (local address 127.30.218.62:39901)
0806 10:53:36.363157 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:36.363164 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:36.363537 (+ 373us) server_negotiation.cc:407] Connection header received
0806 10:53:36.363697 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:36.363701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:36.363781 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:36.363874 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:36.365278 (+ 1404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37518: BlockingRecv error: recv got EOF from 127.0.0.1:37518 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 10:53:36.369016 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 1720
W20260806 10:53:36.373354 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 1720: Not authorized: Client connection negotiation failed: client connection to 127.30.218.1:45029: server requires authentication, but client does not have Kerberos credentials available
I20260806 10:53:36.373631 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 1720
I20260806 10:53:36.390534 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 1869
W20260806 10:53:36.393587 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 1869: Not authorized: Client connection negotiation failed: client connection to 127.30.218.2:34611: server requires authentication, but client does not have Kerberos credentials available
I20260806 10:53:36.393752 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 1869
W20260806 10:53:36.393831 2161 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:53:36.391566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57290 (local address 127.30.218.2:34611)
0806 10:53:36.392216 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:36.392224 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:36.392254 (+ 30us) server_negotiation.cc:407] Connection header received
0806 10:53:36.392323 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:36.392330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:36.392554 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:36.392699 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:36.393520 (+ 821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57290: BlockingRecv error: recv got EOF from 127.0.0.1:57290 (error 108)
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":252,"threads_started":1}
I20260806 10:53:36.410929 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 2012
W20260806 10:53:36.414093 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 2012: Not authorized: Client connection negotiation failed: client connection to 127.30.218.3:45849: server requires authentication, but client does not have Kerberos credentials available
I20260806 10:53:36.414278 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 2012
I20260806 10:53:36.431005 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 1623
I20260806 10:53:36.432363 1687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:36.540737 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 1623
2026-08-06T10:53:36Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4512 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:53:36 dist-test-slave-49h3 krb5kdc[2166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:53:36 dist-test-slave-49h3 krb5kdc[2166](info): set up 2 sockets
Aug 06 10:53:36 dist-test-slave-49h3 krb5kdc[2166](info): commencing operation
krb5kdc: starting...
W20260806 10:53:37.586575 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.005s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:53:37 dist-test-slave-49h3 krb5kdc[2166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:53:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:53:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:53:37.832742 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:35741
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:35741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:38.181309 2180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:38.181722 2180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:38.181881 2180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:38.189677 2180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:53:38.189822 2180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:38.189900 2180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:38.189973 2180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:53:38.190038 2180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:53:38.201047 2180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:35741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:35741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:38.202561 2180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:38.204020 2180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:38.216771 2185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:38.216696 2188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:38.216741 2186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:38.217711 2180 server_base.cc:1034] running on GCE node
I20260806 10:53:38.218556 2180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:38.220098 2180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:38.221323 2180 hybrid_clock.cc:648] HybridClock initialized: now 1786013618221270 us; error 48 us; skew 500 ppm
I20260806 10:53:38.221769 2180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:38 dist-test-slave-49h3 krb5kdc[2166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013618, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:38.228255 2180 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:53:38.230026 2180 webserver.cc:460] Webserver started at http://127.30.218.62:33139/ using document root <none> and password file <none>
I20260806 10:53:38.230804 2180 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:38.230944 2180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:38.231276 2180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:38.234726 2180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "a56bc448b4294a19bff0546cf54ba7d4"
format_stamp: "Formatted at 2026-08-06 10:53:38 on dist-test-slave-49h3"
server_key: "4be5ff293734dc66a9ceaf7bb1c8eddd"
server_key_iv: "9323820b09789360d8b84965c1392c66"
server_key_version: "encryptionkey@0"
I20260806 10:53:38.235600 2180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "a56bc448b4294a19bff0546cf54ba7d4"
format_stamp: "Formatted at 2026-08-06 10:53:38 on dist-test-slave-49h3"
server_key: "4be5ff293734dc66a9ceaf7bb1c8eddd"
server_key_iv: "9323820b09789360d8b84965c1392c66"
server_key_version: "encryptionkey@0"
I20260806 10:53:38.241300 2180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 10:53:38.245067 2195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:38.246531 2180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:53:38.246721 2180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "a56bc448b4294a19bff0546cf54ba7d4"
format_stamp: "Formatted at 2026-08-06 10:53:38 on dist-test-slave-49h3"
server_key: "4be5ff293734dc66a9ceaf7bb1c8eddd"
server_key_iv: "9323820b09789360d8b84965c1392c66"
server_key_version: "encryptionkey@0"
I20260806 10:53:38.246942 2180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:38.278117 2180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:38.282593 2180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:38.282907 2180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:38.302151 2180 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:35741
I20260806 10:53:38.302199 2246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:35741 every 8 connection(s)
I20260806 10:53:38.303637 2180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:53:38.307740 2247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:38.308923 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 2180
I20260806 10:53:38.309163 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:53:38.309866 31592 external_mini_cluster.cc:1428] Setting key 61cfd5031d1ef64c83e485519be2c7f7
I20260806 10:53:38.323165 2247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a56bc448b4294a19bff0546cf54ba7d4: Bootstrap starting.
Aug 06 10:53:38 dist-test-slave-49h3 krb5kdc[2166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:38.326937 2247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a56bc448b4294a19bff0546cf54ba7d4: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:38.328444 2247 log.cc:824] T 00000000000000000000000000000000 P a56bc448b4294a19bff0546cf54ba7d4: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:38.332095 2247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a56bc448b4294a19bff0546cf54ba7d4: No bootstrap required, opened a new log
I20260806 10:53:38.335727 2250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:38.311953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59248 (local address 127.30.218.62:35741)
0806 10:53:38.312929 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:38.312949 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:38.313008 (+ 59us) server_negotiation.cc:407] Connection header received
0806 10:53:38.314798 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:38.314843 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:38.315457 (+ 614us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:38.315975 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:38.317498 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:38.319533 (+ 2035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:38.320934 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:38.322161 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:38.322357 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:38.326717 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:38.326730 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:38.326753 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:38.326818 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:38.330673 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:38.331526 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:38.331535 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:38.331544 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:38.331772 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:38.332132 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:38.332136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:38.332140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:38.332944 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:38.333275 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:38.333697 (+ 422us) server_negotiation.cc:299] Negotiation successful
0806 10:53:38.334229 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":408,"threads_started":1}
I20260806 10:53:38.340096 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:53:38.342065 2247 raft_consensus.cc:348] T 00000000000000000000000000000000 P a56bc448b4294a19bff0546cf54ba7d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a56bc448b4294a19bff0546cf54ba7d4" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 35741 } }
I20260806 10:53:38.342607 2247 raft_consensus.cc:374] T 00000000000000000000000000000000 P a56bc448b4294a19bff0546cf54ba7d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:38.342736 2247 raft_consensus.cc:729] T 00000000000000000000000000000000 P a56bc448b4294a19bff0546cf54ba7d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a56bc448b4294a19bff0546cf54ba7d4, State: Initialized, Role: FOLLOWER
I20260806 10:53:38.343446 2247 consensus_queue.cc:260] T 00000000000000000000000000000000 P a56bc448b4294a19bff0546cf54ba7d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a56bc448b4294a19bff0546cf54ba7d4" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 35741 } }
I20260806 10:53:38.343773 2247 raft_consensus.cc:388] T 00000000000000000000000000000000 P a56bc448b4294a19bff0546cf54ba7d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:53:38.343918 2247 raft_consensus.cc:482] T 00000000000000000000000000000000 P a56bc448b4294a19bff0546cf54ba7d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:53:38.344090 2247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a56bc448b4294a19bff0546cf54ba7d4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:38.347337 2247 raft_consensus.cc:504] T 00000000000000000000000000000000 P a56bc448b4294a19bff0546cf54ba7d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a56bc448b4294a19bff0546cf54ba7d4" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 35741 } }
I20260806 10:53:38.348044 2247 leader_election.cc:302] T 00000000000000000000000000000000 P a56bc448b4294a19bff0546cf54ba7d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a56bc448b4294a19bff0546cf54ba7d4; no voters:
I20260806 10:53:38.348783 2247 leader_election.cc:288] T 00000000000000000000000000000000 P a56bc448b4294a19bff0546cf54ba7d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:53:38.348917 2252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a56bc448b4294a19bff0546cf54ba7d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:38.350203 2252 raft_consensus.cc:686] T 00000000000000000000000000000000 P a56bc448b4294a19bff0546cf54ba7d4 [term 1 LEADER]: Becoming Leader. State: Replica: a56bc448b4294a19bff0546cf54ba7d4, State: Running, Role: LEADER
I20260806 10:53:38.350860 2252 consensus_queue.cc:237] T 00000000000000000000000000000000 P a56bc448b4294a19bff0546cf54ba7d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a56bc448b4294a19bff0546cf54ba7d4" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 35741 } }
I20260806 10:53:38.351238 2247 sys_catalog.cc:564] T 00000000000000000000000000000000 P a56bc448b4294a19bff0546cf54ba7d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:53:38.354317 2253 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56bc448b4294a19bff0546cf54ba7d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a56bc448b4294a19bff0546cf54ba7d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a56bc448b4294a19bff0546cf54ba7d4" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 35741 } } }
I20260806 10:53:38.354218 2254 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56bc448b4294a19bff0546cf54ba7d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a56bc448b4294a19bff0546cf54ba7d4. Latest consensus state: current_term: 1 leader_uuid: "a56bc448b4294a19bff0546cf54ba7d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a56bc448b4294a19bff0546cf54ba7d4" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 35741 } } }
I20260806 10:53:38.354722 2253 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56bc448b4294a19bff0546cf54ba7d4 [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:38.354717 2254 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56bc448b4294a19bff0546cf54ba7d4 [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:38.368011 2261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:53:38.373832 2261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:53:38.384042 2261 catalog_manager.cc:1347] Generated new cluster ID: 7ffef986ea93408c8fb1e413186a51c2
I20260806 10:53:38.384187 2261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:53:38.400779 2250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:38.392916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59256 (local address 127.30.218.62:35741)
0806 10:53:38.393151 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:38.393159 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:38.393755 (+ 596us) server_negotiation.cc:407] Connection header received
0806 10:53:38.393950 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:38.393955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:38.394026 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:38.394114 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:38.395250 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:38.396027 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:38.397394 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:38.397910 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:38.398045 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:38.399185 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:38.399192 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:38.399196 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:53:38.399905 (+ 709us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:38.399981 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:53:38.400014 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:38.400127 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:38.400269 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 10:53:38.400398 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 10:53:38.412045 2261 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:53:38.413638 2261 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:53:38.422446 2261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a56bc448b4294a19bff0546cf54ba7d4: Generated new TSK 0
I20260806 10:53:38.423395 2261 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:53:38.432546 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 2180
I20260806 10:53:38.433809 2242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:38.527499 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 2180
2026-08-06T10:53:38Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1980 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:53:38 dist-test-slave-49h3 krb5kdc[2281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:53:38 dist-test-slave-49h3 krb5kdc[2281](info): set up 2 sockets
Aug 06 10:53:38 dist-test-slave-49h3 krb5kdc[2281](info): commencing operation
krb5kdc: starting...
W20260806 10:53:39.567101 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:53:39 dist-test-slave-49h3 krb5kdc[2281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:53:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:53:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:53:39.810760 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:42321
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:42321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:40.151420 2295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:40.151767 2295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:40.151894 2295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:40.159396 2295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:53:40.159545 2295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:40.159627 2295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:40.159709 2295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:53:40.159773 2295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:53:40.170769 2295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:42321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:42321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:40.172260 2295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:40.173694 2295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:40.186494 2300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:40.186589 2303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:40.186533 2301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:40.187013 2295 server_base.cc:1034] running on GCE node
I20260806 10:53:40.188129 2295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:40.189699 2295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:40.190955 2295 hybrid_clock.cc:648] HybridClock initialized: now 1786013620190892 us; error 45 us; skew 500 ppm
I20260806 10:53:40.191366 2295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:40 dist-test-slave-49h3 krb5kdc[2281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013620, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:40.194765 2295 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:53:40.196403 2295 webserver.cc:460] Webserver started at http://127.30.218.62:35663/ using document root <none> and password file <none>
I20260806 10:53:40.197137 2295 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:40.197266 2295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:40.197618 2295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:40.200897 2295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "4a36853b6679471b86a0cc728ebab2b5"
format_stamp: "Formatted at 2026-08-06 10:53:40 on dist-test-slave-49h3"
server_key: "cbec0f09dabf6c995f6b4e50ffd15cc0"
server_key_iv: "faaf6e9a6d271c67c57a4a13d84e680f"
server_key_version: "encryptionkey@0"
I20260806 10:53:40.201799 2295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "4a36853b6679471b86a0cc728ebab2b5"
format_stamp: "Formatted at 2026-08-06 10:53:40 on dist-test-slave-49h3"
server_key: "cbec0f09dabf6c995f6b4e50ffd15cc0"
server_key_iv: "faaf6e9a6d271c67c57a4a13d84e680f"
server_key_version: "encryptionkey@0"
I20260806 10:53:40.207595 2295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:53:40.211700 2310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:40.213393 2295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:53:40.213647 2295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "4a36853b6679471b86a0cc728ebab2b5"
format_stamp: "Formatted at 2026-08-06 10:53:40 on dist-test-slave-49h3"
server_key: "cbec0f09dabf6c995f6b4e50ffd15cc0"
server_key_iv: "faaf6e9a6d271c67c57a4a13d84e680f"
server_key_version: "encryptionkey@0"
I20260806 10:53:40.213876 2295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:40.230440 2295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:40.234827 2295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:40.235150 2295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:40.254009 2295 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:42321
I20260806 10:53:40.254041 2361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:42321 every 8 connection(s)
I20260806 10:53:40.255589 2295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:53:40.258260 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 2295
I20260806 10:53:40.258492 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:53:40.259142 31592 external_mini_cluster.cc:1428] Setting key e1c62523f09546b37541647ad5fb76ea
I20260806 10:53:40.259766 2362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:40.274976 2362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a36853b6679471b86a0cc728ebab2b5: Bootstrap starting.
Aug 06 10:53:40 dist-test-slave-49h3 krb5kdc[2281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:40.279145 2362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a36853b6679471b86a0cc728ebab2b5: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:40.280572 2362 log.cc:824] T 00000000000000000000000000000000 P 4a36853b6679471b86a0cc728ebab2b5: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:40.284134 2362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a36853b6679471b86a0cc728ebab2b5: No bootstrap required, opened a new log
I20260806 10:53:40.285152 2365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:40.261387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53888 (local address 127.30.218.62:42321)
0806 10:53:40.262308 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:40.262325 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:40.262390 (+ 65us) server_negotiation.cc:407] Connection header received
0806 10:53:40.264914 (+ 2524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:40.264951 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:40.265612 (+ 661us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:40.266161 (+ 549us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:40.267561 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:40.269601 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:40.270932 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:40.272076 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:40.272275 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:40.276470 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:40.276481 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:40.276501 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:40.276572 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:40.280520 (+ 3948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:40.281320 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:40.281333 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:40.281339 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:40.281555 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:40.281986 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:40.281990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:40.281993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:40.282709 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:40.283009 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:40.283320 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 10:53:40.283771 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":390,"threads_started":1}
I20260806 10:53:40.293030 2362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a36853b6679471b86a0cc728ebab2b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a36853b6679471b86a0cc728ebab2b5" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42321 } }
I20260806 10:53:40.293435 2362 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a36853b6679471b86a0cc728ebab2b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:40.293548 2362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a36853b6679471b86a0cc728ebab2b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a36853b6679471b86a0cc728ebab2b5, State: Initialized, Role: FOLLOWER
I20260806 10:53:40.294090 2362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a36853b6679471b86a0cc728ebab2b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a36853b6679471b86a0cc728ebab2b5" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42321 } }
I20260806 10:53:40.294313 2362 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a36853b6679471b86a0cc728ebab2b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:53:40.294413 2362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a36853b6679471b86a0cc728ebab2b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:53:40.294534 2362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a36853b6679471b86a0cc728ebab2b5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:40.297132 2362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a36853b6679471b86a0cc728ebab2b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a36853b6679471b86a0cc728ebab2b5" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42321 } }
I20260806 10:53:40.297716 2362 leader_election.cc:302] T 00000000000000000000000000000000 P 4a36853b6679471b86a0cc728ebab2b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a36853b6679471b86a0cc728ebab2b5; no voters:
I20260806 10:53:40.298298 2362 leader_election.cc:288] T 00000000000000000000000000000000 P 4a36853b6679471b86a0cc728ebab2b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:53:40.298419 2367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a36853b6679471b86a0cc728ebab2b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:40.299084 2367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a36853b6679471b86a0cc728ebab2b5 [term 1 LEADER]: Becoming Leader. State: Replica: 4a36853b6679471b86a0cc728ebab2b5, State: Running, Role: LEADER
I20260806 10:53:40.299683 2367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a36853b6679471b86a0cc728ebab2b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a36853b6679471b86a0cc728ebab2b5" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42321 } }
I20260806 10:53:40.300549 2362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a36853b6679471b86a0cc728ebab2b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:53:40.302574 2369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a36853b6679471b86a0cc728ebab2b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a36853b6679471b86a0cc728ebab2b5. Latest consensus state: current_term: 1 leader_uuid: "4a36853b6679471b86a0cc728ebab2b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a36853b6679471b86a0cc728ebab2b5" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42321 } } }
I20260806 10:53:40.302541 2368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a36853b6679471b86a0cc728ebab2b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a36853b6679471b86a0cc728ebab2b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a36853b6679471b86a0cc728ebab2b5" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42321 } } }
I20260806 10:53:40.302946 2369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a36853b6679471b86a0cc728ebab2b5 [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:40.302946 2368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a36853b6679471b86a0cc728ebab2b5 [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:40.306248 2374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:53:40.310667 2374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:53:40.320061 2374 catalog_manager.cc:1347] Generated new cluster ID: 2e70755ddb3a4361aff878b394fb2625
I20260806 10:53:40.320192 2374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:53:40.339108 2374 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:53:40.340452 2374 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:53:40.357483 2374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a36853b6679471b86a0cc728ebab2b5: Generated new TSK 0
I20260806 10:53:40.358557 2374 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
I20260806 10:53:40.423277 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:42321
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:40.773581 2389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:40.773888 2389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:40.774019 2389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:40.781824 2389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:40.781963 2389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:40.782159 2389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:53:40.792646 2389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:42321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:40.794135 2389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:40.795485 2389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:40.808915 2394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:40.808964 2395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:40.809161 2397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:40.810010 2389 server_base.cc:1034] running on GCE node
I20260806 10:53:40.810644 2389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:40.811579 2389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:40.812800 2389 hybrid_clock.cc:648] HybridClock initialized: now 1786013620812740 us; error 50 us; skew 500 ppm
I20260806 10:53:40.813118 2389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:40 dist-test-slave-49h3 krb5kdc[2281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013620, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:40.816249 2389 init.cc:377] Logged in from keytab as kudu/127.30.218.1@KRBTEST.COM (short username kudu)
I20260806 10:53:40.817837 2389 webserver.cc:460] Webserver started at http://127.30.218.1:39717/ using document root <none> and password file <none>
I20260806 10:53:40.818626 2389 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:40.818742 2389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:40.819051 2389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:40.822328 2389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "469809f8056049fb898ebb94c91162fe"
format_stamp: "Formatted at 2026-08-06 10:53:40 on dist-test-slave-49h3"
server_key: "cb038b148daa94bc3197f52e479ae549"
server_key_iv: "a112029a9892a26fb60621bae2e76a7a"
server_key_version: "encryptionkey@0"
I20260806 10:53:40.823174 2389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "469809f8056049fb898ebb94c91162fe"
format_stamp: "Formatted at 2026-08-06 10:53:40 on dist-test-slave-49h3"
server_key: "cb038b148daa94bc3197f52e479ae549"
server_key_iv: "a112029a9892a26fb60621bae2e76a7a"
server_key_version: "encryptionkey@0"
I20260806 10:53:40.828926 2389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 10:53:40.832805 2404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:40.834266 2389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 10:53:40.834422 2389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "469809f8056049fb898ebb94c91162fe"
format_stamp: "Formatted at 2026-08-06 10:53:40 on dist-test-slave-49h3"
server_key: "cb038b148daa94bc3197f52e479ae549"
server_key_iv: "a112029a9892a26fb60621bae2e76a7a"
server_key_version: "encryptionkey@0"
I20260806 10:53:40.834627 2389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:40.862953 2389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:40.867054 2389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:40.867349 2389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:40.868330 2389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:53:40.869678 2389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:40.869751 2389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:40.869868 2389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:40.869920 2389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:40.901515 2389 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:46665
I20260806 10:53:40.901582 2518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:46665 every 8 connection(s)
I20260806 10:53:40.902967 2389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:53:40.909070 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 2389
I20260806 10:53:40.909314 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:53:40.909992 31592 external_mini_cluster.cc:1428] Setting key e129a13ea780be961bbddf046db0cf63
Aug 06 10:53:40 dist-test-slave-49h3 krb5kdc[2281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013620, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:40.925405 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:40.905707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:33851 (local address 127.30.218.62:42321)
0806 10:53:40.906177 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:40.906182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:40.907338 (+ 1156us) server_negotiation.cc:407] Connection header received
0806 10:53:40.909363 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:40.909369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:40.909495 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:40.909603 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:40.911455 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:40.912352 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:40.914223 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:40.914817 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:40.914956 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:40.918796 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:40.918803 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:40.918807 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:40.918849 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:40.921827 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:40.922795 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:40.922799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:40.922801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:40.922932 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:40.923407 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:40.923410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:40.923412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:40.923700 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:40.923823 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:40.924911 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0806 10:53:40.925123 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":217,"threads_started":1}
I20260806 10:53:40.926894 2522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:40.906147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42321 (local address 127.30.218.1:33851)
0806 10:53:40.907150 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:40.907192 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:40.909044 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:40.909876 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:40.909886 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:40.910463 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:40.911246 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:40.911273 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:40.912634 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:40.912640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:40.914050 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:40.914064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:40.915021 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:40.915026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:40.915247 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:40.915937 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:40.915962 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:40.918541 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:40.922001 (+ 3460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:40.922012 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:40.922032 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:40.922659 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:40.923031 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:40.923036 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:40.923040 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:40.923310 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:40.923825 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:40.923835 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:40.924351 (+ 516us) client_negotiation.cc:769] Sending connection context
0806 10:53:40.924706 (+ 355us) client_negotiation.cc:240] Negotiation successful
0806 10:53:40.925357 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":518,"threads_started":1}
I20260806 10:53:40.928627 2519 heartbeater.cc:347] Connected to a master server at 127.30.218.62:42321
I20260806 10:53:40.929097 2519 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:40.930182 2519 heartbeater.cc:511] Master 127.30.218.62:42321 requested a full tablet report, sending...
I20260806 10:53:40.932691 2327 ts_manager.cc:194] Registered new tserver with Master: 469809f8056049fb898ebb94c91162fe (127.30.218.1:46665)
I20260806 10:53:40.934288 2327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.1@KRBTEST.COM'} at 127.30.218.1:33851
WARNING: no policy specified for kudu/127.30.218.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
I20260806 10:53:40.965193 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:42321
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:41.303785 2525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:41.304075 2525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:41.304219 2525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:41.311426 2525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:41.311548 2525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:41.311703 2525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:53:41.321798 2525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:42321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:41.323123 2525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:41.324404 2525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:41.336932 2530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:41.336952 2533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:41.336921 2531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:41.337647 2525 server_base.cc:1034] running on GCE node
I20260806 10:53:41.338359 2525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:41.339298 2525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:41.340520 2525 hybrid_clock.cc:648] HybridClock initialized: now 1786013621340470 us; error 43 us; skew 500 ppm
I20260806 10:53:41.340833 2525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:41 dist-test-slave-49h3 krb5kdc[2281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013621, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:41.343905 2525 init.cc:377] Logged in from keytab as kudu/127.30.218.2@KRBTEST.COM (short username kudu)
I20260806 10:53:41.345371 2525 webserver.cc:460] Webserver started at http://127.30.218.2:36433/ using document root <none> and password file <none>
I20260806 10:53:41.346081 2525 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:41.346185 2525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:41.346525 2525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:41.349391 2525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "76d7a97afeac4dd0b3a79773306d3b94"
format_stamp: "Formatted at 2026-08-06 10:53:41 on dist-test-slave-49h3"
server_key: "6bfa8d4f53a39cb239c96efcf87c4154"
server_key_iv: "c181fe6b4b9f48f2207194379d6befc2"
server_key_version: "encryptionkey@0"
I20260806 10:53:41.350158 2525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "76d7a97afeac4dd0b3a79773306d3b94"
format_stamp: "Formatted at 2026-08-06 10:53:41 on dist-test-slave-49h3"
server_key: "6bfa8d4f53a39cb239c96efcf87c4154"
server_key_iv: "c181fe6b4b9f48f2207194379d6befc2"
server_key_version: "encryptionkey@0"
I20260806 10:53:41.355156 2525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:53:41.358700 2540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:41.360061 2525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:53:41.360211 2525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "76d7a97afeac4dd0b3a79773306d3b94"
format_stamp: "Formatted at 2026-08-06 10:53:41 on dist-test-slave-49h3"
server_key: "6bfa8d4f53a39cb239c96efcf87c4154"
server_key_iv: "c181fe6b4b9f48f2207194379d6befc2"
server_key_version: "encryptionkey@0"
I20260806 10:53:41.360385 2525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:41.377665 2525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:41.382266 2525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:41.382545 2525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:41.383673 2525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:53:41.385193 2525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:41.385275 2525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:41.385367 2525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:41.385411 2525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:41.417294 2525 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:37533
I20260806 10:53:41.417327 2652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:37533 every 8 connection(s)
I20260806 10:53:41.418859 2525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:53:41.419548 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 2525
I20260806 10:53:41.419762 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:53:41.420337 31592 external_mini_cluster.cc:1428] Setting key 41d0a7657989b69813e344d6d2566b7e
Aug 06 10:53:41 dist-test-slave-49h3 krb5kdc[2281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013621, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:41.439994 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:41.421665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:34661 (local address 127.30.218.62:42321)
0806 10:53:41.421876 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:41.421882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:41.423236 (+ 1354us) server_negotiation.cc:407] Connection header received
0806 10:53:41.425103 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:41.425109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:41.425193 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:41.425283 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:41.426933 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:41.427780 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:41.429398 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:41.429889 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:41.430012 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:41.433977 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:41.433982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:41.433985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:41.434022 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:41.436656 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:41.437509 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:41.437513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:41.437516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:41.437661 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:41.438121 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:41.438125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:41.438127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:41.438475 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:41.438634 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:41.439368 (+ 734us) server_negotiation.cc:299] Negotiation successful
0806 10:53:41.439632 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 10:53:41.441087 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:41.422256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42321 (local address 127.30.218.2:34661)
0806 10:53:41.423071 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:41.423117 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:41.424802 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:41.425598 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:41.425607 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:41.426160 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:41.426756 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:41.426775 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:41.427951 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:41.427955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:41.429245 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:41.429259 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:41.430136 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:41.430144 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:41.430398 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:41.431123 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:41.431154 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:41.433718 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:41.436833 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:41.436844 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:41.436860 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:41.437340 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:41.437777 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:41.437783 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:41.437787 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:41.437995 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:41.438611 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:41.438619 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:41.438989 (+ 370us) client_negotiation.cc:769] Sending connection context
0806 10:53:41.439249 (+ 260us) client_negotiation.cc:240] Negotiation successful
0806 10:53:41.439909 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":335,"threads_started":1}
WARNING: no policy specified for kudu/127.30.218.3@KRBTEST.COM; defaulting to no policy
I20260806 10:53:41.442736 2653 heartbeater.cc:347] Connected to a master server at 127.30.218.62:42321
I20260806 10:53:41.443040 2653 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:41.443826 2653 heartbeater.cc:511] Master 127.30.218.62:42321 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.3@KRBTEST.COM" created.
I20260806 10:53:41.445518 2327 ts_manager.cc:194] Registered new tserver with Master: 76d7a97afeac4dd0b3a79773306d3b94 (127.30.218.2:37533)
I20260806 10:53:41.446398 2327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.2@KRBTEST.COM'} at 127.30.218.2:34661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
I20260806 10:53:41.472401 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:42321
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:41.815438 2658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:41.815784 2658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:41.815899 2658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:41.823851 2658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:41.824000 2658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:41.824131 2658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:53:41.834637 2658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:42321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:41.836068 2658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:41.837409 2658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:41.851136 2663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:41.851111 2666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:41.851065 2664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:41.851780 2658 server_base.cc:1034] running on GCE node
I20260806 10:53:41.852449 2658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:41.853399 2658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:41.854624 2658 hybrid_clock.cc:648] HybridClock initialized: now 1786013621854580 us; error 40 us; skew 500 ppm
I20260806 10:53:41.854936 2658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:41 dist-test-slave-49h3 krb5kdc[2281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013621, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:41.858141 2658 init.cc:377] Logged in from keytab as kudu/127.30.218.3@KRBTEST.COM (short username kudu)
I20260806 10:53:41.859512 2658 webserver.cc:460] Webserver started at http://127.30.218.3:36215/ using document root <none> and password file <none>
I20260806 10:53:41.860167 2658 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:41.860296 2658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:41.860597 2658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:41.863675 2658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "7dcb17a796ca4e748764575b004c52e6"
format_stamp: "Formatted at 2026-08-06 10:53:41 on dist-test-slave-49h3"
server_key: "b193032186c1193a23ce977c9b77bb34"
server_key_iv: "93fe6681f9654bf8a281acbdf7099cae"
server_key_version: "encryptionkey@0"
I20260806 10:53:41.864552 2658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "7dcb17a796ca4e748764575b004c52e6"
format_stamp: "Formatted at 2026-08-06 10:53:41 on dist-test-slave-49h3"
server_key: "b193032186c1193a23ce977c9b77bb34"
server_key_iv: "93fe6681f9654bf8a281acbdf7099cae"
server_key_version: "encryptionkey@0"
I20260806 10:53:41.870388 2658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 10:53:41.874231 2673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:41.875743 2658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:53:41.875917 2658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "7dcb17a796ca4e748764575b004c52e6"
format_stamp: "Formatted at 2026-08-06 10:53:41 on dist-test-slave-49h3"
server_key: "b193032186c1193a23ce977c9b77bb34"
server_key_iv: "93fe6681f9654bf8a281acbdf7099cae"
server_key_version: "encryptionkey@0"
I20260806 10:53:41.876121 2658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:41.920730 2658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:41.925089 2658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:41.925400 2658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:41.926477 2658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:53:41.927865 2658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:41.927944 2658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:41.928073 2658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:41.928130 2658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:41.937825 2519 heartbeater.cc:503] Master 127.30.218.62:42321 was elected leader, sending a full tablet report...
I20260806 10:53:41.963320 2658 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:45889
I20260806 10:53:41.963364 2785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:45889 every 8 connection(s)
I20260806 10:53:41.964705 2658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:53:41.967695 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 2658
I20260806 10:53:41.967909 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:53:41.968559 31592 external_mini_cluster.cc:1428] Setting key 9bb9290baceb331009e4bd56b15d911e
Aug 06 10:53:41 dist-test-slave-49h3 krb5kdc[2281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013621, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:41.985782 2788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:41.967407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:41117 (local address 127.30.218.62:42321)
0806 10:53:41.967843 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:41.967850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:41.968982 (+ 1132us) server_negotiation.cc:407] Connection header received
0806 10:53:41.971118 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:41.971124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:41.971212 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:41.971311 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:41.972942 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:41.973795 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:41.975352 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:41.975749 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:41.975866 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:41.979444 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:41.979449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:41.979452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:41.979502 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:41.982072 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:41.982946 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:41.982951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:41.982954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:41.983126 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:41.983655 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:41.983659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:41.983663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:41.983999 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:41.984196 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:41.985088 (+ 892us) server_negotiation.cc:299] Negotiation successful
0806 10:53:41.985344 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":212,"threads_started":1}
I20260806 10:53:41.986832 2789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:41.967971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42321 (local address 127.30.218.3:41117)
0806 10:53:41.968814 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:41.968858 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:41.970845 (+ 1987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:41.971590 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:41.971601 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:41.972177 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:41.972762 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:41.972780 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:41.973942 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:41.973947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:41.975203 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:41.975216 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:41.975977 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:41.975985 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:41.976222 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:41.976894 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:41.976918 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:41.979197 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:41.982231 (+ 3034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:41.982241 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:41.982261 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:41.982814 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:41.983255 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:41.983258 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:41.983261 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:41.983541 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:41.984261 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:41.984268 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:41.984689 (+ 421us) client_negotiation.cc:769] Sending connection context
0806 10:53:41.984967 (+ 278us) client_negotiation.cc:240] Negotiation successful
0806 10:53:41.985597 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":341,"threads_started":1}
I20260806 10:53:41.988251 2786 heartbeater.cc:347] Connected to a master server at 127.30.218.62:42321
I20260806 10:53:41.988562 2786 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:41.989327 2786 heartbeater.cc:511] Master 127.30.218.62:42321 requested a full tablet report, sending...
I20260806 10:53:41.990876 2327 ts_manager.cc:194] Registered new tserver with Master: 7dcb17a796ca4e748764575b004c52e6 (127.30.218.3:45889)
I20260806 10:53:41.991730 2327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.3@KRBTEST.COM'} at 127.30.218.3:41117
I20260806 10:53:41.996809 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 10:53:42 dist-test-slave-49h3 krb5kdc[2281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013622, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 10:53:42 dist-test-slave-49h3 krb5kdc[2281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013622, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:42.056376 2788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:42.041627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53898 (local address 127.30.218.62:42321)
0806 10:53:42.041802 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:42.041807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:42.042214 (+ 407us) server_negotiation.cc:407] Connection header received
0806 10:53:42.042376 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:42.042381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:42.042453 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:42.042548 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:42.043957 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:42.044712 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:42.046148 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:42.046586 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:42.046709 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:42.050943 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:42.050949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:42.050952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:42.050999 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:42.053785 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:42.054551 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:42.054555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:42.054557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:42.054691 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:42.055076 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:42.055079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:42.055088 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:42.055442 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:42.055632 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:42.055913 (+ 281us) server_negotiation.cc:299] Negotiation successful
0806 10:53:42.056126 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 10:53:42.057026 2327 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53898
I20260806 10:53:42.058957 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 2389
Aug 06 10:53:42 dist-test-slave-49h3 krb5kdc[2281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013622, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.30.218.1@KRBTEST.COM
I20260806 10:53:42.076056 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:42.060248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42242 (local address 127.30.218.1:46665)
0806 10:53:42.060924 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:42.060931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:42.060958 (+ 27us) server_negotiation.cc:407] Connection header received
0806 10:53:42.061019 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:42.061027 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:42.061266 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:42.061398 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:42.062716 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:42.063891 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:42.065338 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:42.066033 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:42.066140 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:42.070233 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:42.070249 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:42.070265 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:42.070314 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:42.073226 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:42.074053 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:42.074062 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:42.074064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:42.074231 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:42.074647 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:42.074650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:42.074652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:42.075036 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:42.075277 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:42.075523 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 10:53:42.075758 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":350,"threads_started":1}
W20260806 10:53:42.077355 2514 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42242
W20260806 10:53:42.078516 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 2389: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:53:42.078725 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 2389
I20260806 10:53:42.095773 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 2525
Aug 06 10:53:42 dist-test-slave-49h3 krb5kdc[2281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013622, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.30.218.2@KRBTEST.COM
I20260806 10:53:42.112767 2799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:42.096723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43064 (local address 127.30.218.2:37533)
0806 10:53:42.097306 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:42.097314 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:42.097342 (+ 28us) server_negotiation.cc:407] Connection header received
0806 10:53:42.097407 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:42.097414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:42.097694 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:42.097838 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:42.099213 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:42.100271 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:42.101696 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:42.102346 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:42.102463 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:42.106841 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:42.106857 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:42.106872 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:42.106913 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:42.109925 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:42.110593 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:42.110600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:42.110603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:42.110732 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:42.111116 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:42.111119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:42.111121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:42.111492 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:42.111789 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:42.112123 (+ 334us) server_negotiation.cc:299] Negotiation successful
0806 10:53:42.112409 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":319,"threads_started":1}
W20260806 10:53:42.114074 2648 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43064
W20260806 10:53:42.115118 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 2525: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:53:42.115296 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 2525
I20260806 10:53:42.131471 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 2658
Aug 06 10:53:42 dist-test-slave-49h3 krb5kdc[2281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013622, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.30.218.3@KRBTEST.COM
I20260806 10:53:42.149256 2800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:42.132487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55578 (local address 127.30.218.3:45889)
0806 10:53:42.133115 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:42.133122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:42.133163 (+ 41us) server_negotiation.cc:407] Connection header received
0806 10:53:42.133229 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:42.133236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:42.133495 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:42.133633 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:42.134988 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:42.136084 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:42.137658 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:42.138276 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:42.138381 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:42.142919 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:42.142939 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:42.142957 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:42.142999 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:42.146151 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:42.146951 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:42.146959 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:42.146962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:42.147101 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:42.147549 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:42.147553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:42.147556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:42.147973 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:42.148274 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:42.148604 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 10:53:42.148855 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":301,"threads_started":1}
W20260806 10:53:42.150823 2781 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55578
W20260806 10:53:42.152109 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 2658: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:53:42.152287 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 2658
I20260806 10:53:42.168310 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 2295
I20260806 10:53:42.169674 2357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:42.275925 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 2295
2026-08-06T10:53:42Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3753 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:53:42 dist-test-slave-49h3 krb5kdc[2804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:53:42 dist-test-slave-49h3 krb5kdc[2804](info): set up 2 sockets
Aug 06 10:53:42 dist-test-slave-49h3 krb5kdc[2804](info): commencing operation
krb5kdc: starting...
W20260806 10:53:43.327108 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.005s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:53:43 dist-test-slave-49h3 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:53:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:53:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:53:43.570745 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:41915
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:37775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:41915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:43.916110 2817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:43.916452 2817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:43.916571 2817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:43.924083 2817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:53:43.924214 2817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:43.924275 2817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:43.924355 2817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:53:43.924409 2817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:53:43.936105 2817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:37775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:41915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:41915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:43.938573 2817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:43.940655 2817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:43.955129 2822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:43.955066 2825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:43.955196 2823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:43.955936 2817 server_base.cc:1034] running on GCE node
I20260806 10:53:43.956796 2817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:43.958485 2817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:43.959692 2817 hybrid_clock.cc:648] HybridClock initialized: now 1786013623959645 us; error 34 us; skew 500 ppm
I20260806 10:53:43.960060 2817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:43 dist-test-slave-49h3 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013623, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:43.967031 2817 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:53:43.968961 2817 webserver.cc:460] Webserver started at http://127.30.218.62:32919/ using document root <none> and password file <none>
I20260806 10:53:43.969848 2817 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:43.969966 2817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:43.970330 2817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:43.974287 2817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "9e69dc50c14340779370d450888d369d"
format_stamp: "Formatted at 2026-08-06 10:53:43 on dist-test-slave-49h3"
server_key: "80f10b4665d1f62bfec9f57a8b93e861"
server_key_iv: "c07e92e39f38f6c7863cb97981f9b5e4"
server_key_version: "encryptionkey@0"
I20260806 10:53:43.975317 2817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "9e69dc50c14340779370d450888d369d"
format_stamp: "Formatted at 2026-08-06 10:53:43 on dist-test-slave-49h3"
server_key: "80f10b4665d1f62bfec9f57a8b93e861"
server_key_iv: "c07e92e39f38f6c7863cb97981f9b5e4"
server_key_version: "encryptionkey@0"
I20260806 10:53:43.982141 2817 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 10:53:43.986670 2832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:43.988387 2817 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 10:53:43.988586 2817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "9e69dc50c14340779370d450888d369d"
format_stamp: "Formatted at 2026-08-06 10:53:43 on dist-test-slave-49h3"
server_key: "80f10b4665d1f62bfec9f57a8b93e861"
server_key_iv: "c07e92e39f38f6c7863cb97981f9b5e4"
server_key_version: "encryptionkey@0"
I20260806 10:53:43.988822 2817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:44.013543 2817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:44.018507 2817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:44.018790 2817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:44.038910 2817 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:41915
I20260806 10:53:44.038918 2883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:41915 every 8 connection(s)
I20260806 10:53:44.040650 2817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:53:44.044989 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 2817
I20260806 10:53:44.045228 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:53:44.045573 2884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:44.046092 31592 external_mini_cluster.cc:1428] Setting key aadb216c4ffbdc01d4e3df50a1b9c24b
I20260806 10:53:44.063134 2884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e69dc50c14340779370d450888d369d: Bootstrap starting.
Aug 06 10:53:44 dist-test-slave-49h3 krb5kdc[2804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:44.068121 2884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e69dc50c14340779370d450888d369d: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:44.069693 2884 log.cc:824] T 00000000000000000000000000000000 P 9e69dc50c14340779370d450888d369d: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:44.073650 2884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e69dc50c14340779370d450888d369d: No bootstrap required, opened a new log
I20260806 10:53:44.073868 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:44.048466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60500 (local address 127.30.218.62:41915)
0806 10:53:44.049412 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:44.049426 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:44.049520 (+ 94us) server_negotiation.cc:407] Connection header received
0806 10:53:44.051343 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:44.051374 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:44.051938 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:44.052395 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:44.054778 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:44.056219 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:44.058157 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:44.059015 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:44.059171 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:44.064440 (+ 5269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:44.064453 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:44.064478 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:44.064541 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:44.068755 (+ 4214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:44.069613 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:44.069623 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:44.069634 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:44.069909 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:44.070302 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:44.070318 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:44.070322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:44.071064 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:44.071386 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:44.071821 (+ 435us) server_negotiation.cc:299] Negotiation successful
0806 10:53:44.072324 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":384,"threads_started":1}
I20260806 10:53:44.082134 2884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e69dc50c14340779370d450888d369d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e69dc50c14340779370d450888d369d" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41915 } }
I20260806 10:53:44.082556 2884 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e69dc50c14340779370d450888d369d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:44.082638 2884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e69dc50c14340779370d450888d369d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e69dc50c14340779370d450888d369d, State: Initialized, Role: FOLLOWER
I20260806 10:53:44.083245 2884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e69dc50c14340779370d450888d369d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e69dc50c14340779370d450888d369d" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41915 } }
I20260806 10:53:44.083485 2884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e69dc50c14340779370d450888d369d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:53:44.083595 2884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e69dc50c14340779370d450888d369d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:53:44.083729 2884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e69dc50c14340779370d450888d369d [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:44.086251 2884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e69dc50c14340779370d450888d369d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e69dc50c14340779370d450888d369d" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41915 } }
I20260806 10:53:44.086896 2884 leader_election.cc:302] T 00000000000000000000000000000000 P 9e69dc50c14340779370d450888d369d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e69dc50c14340779370d450888d369d; no voters:
I20260806 10:53:44.087568 2884 leader_election.cc:288] T 00000000000000000000000000000000 P 9e69dc50c14340779370d450888d369d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:53:44.087705 2889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e69dc50c14340779370d450888d369d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:44.088449 2889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e69dc50c14340779370d450888d369d [term 1 LEADER]: Becoming Leader. State: Replica: 9e69dc50c14340779370d450888d369d, State: Running, Role: LEADER
I20260806 10:53:44.089108 2889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e69dc50c14340779370d450888d369d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e69dc50c14340779370d450888d369d" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41915 } }
I20260806 10:53:44.090109 2884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e69dc50c14340779370d450888d369d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:53:44.092355 2891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e69dc50c14340779370d450888d369d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e69dc50c14340779370d450888d369d. Latest consensus state: current_term: 1 leader_uuid: "9e69dc50c14340779370d450888d369d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e69dc50c14340779370d450888d369d" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41915 } } }
I20260806 10:53:44.092844 2890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e69dc50c14340779370d450888d369d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e69dc50c14340779370d450888d369d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e69dc50c14340779370d450888d369d" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41915 } } }
I20260806 10:53:44.093391 2890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e69dc50c14340779370d450888d369d [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:44.093801 2891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e69dc50c14340779370d450888d369d [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:44.097771 2895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:53:44.104257 2895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:53:44.116092 2895 catalog_manager.cc:1347] Generated new cluster ID: 96ac8911fc1d4d7b9c5b568c5881474a
I20260806 10:53:44.116264 2895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:53:44.127926 2895 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:53:44.130396 2895 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:53:44.138393 2895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e69dc50c14340779370d450888d369d: Generated new TSK 0
I20260806 10:53:44.139719 2895 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
I20260806 10:53:44.211913 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:41915
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:37775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:44.603608 2910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:44.603986 2910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:44.604136 2910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:44.612205 2910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:44.612371 2910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:44.612546 2910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:53:44.624128 2910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:37775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:41915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:44.626049 2910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:44.627770 2910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:44.643718 2915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:44.643649 2916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:44.643677 2918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:44.645210 2910 server_base.cc:1034] running on GCE node
I20260806 10:53:44.646142 2910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:44.647419 2910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:44.648653 2910 hybrid_clock.cc:648] HybridClock initialized: now 1786013624648592 us; error 50 us; skew 500 ppm
I20260806 10:53:44.649024 2910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:44 dist-test-slave-49h3 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013624, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:44.653028 2910 init.cc:377] Logged in from keytab as kudu/127.30.218.1@KRBTEST.COM (short username kudu)
I20260806 10:53:44.654901 2910 webserver.cc:460] Webserver started at http://127.30.218.1:34911/ using document root <none> and password file <none>
I20260806 10:53:44.655942 2910 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:44.656097 2910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:44.656513 2910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:44.660401 2910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "507915a50f16444aa3c39a07c3779454"
format_stamp: "Formatted at 2026-08-06 10:53:44 on dist-test-slave-49h3"
server_key: "a4d3ac0445968a9e323fbb690a98b73b"
server_key_iv: "982bec2d25b441efe92b5d70c4afc507"
server_key_version: "encryptionkey@0"
I20260806 10:53:44.661418 2910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "507915a50f16444aa3c39a07c3779454"
format_stamp: "Formatted at 2026-08-06 10:53:44 on dist-test-slave-49h3"
server_key: "a4d3ac0445968a9e323fbb690a98b73b"
server_key_iv: "982bec2d25b441efe92b5d70c4afc507"
server_key_version: "encryptionkey@0"
I20260806 10:53:44.667996 2910 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260806 10:53:44.672183 2925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:44.673951 2910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 10:53:44.674129 2910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "507915a50f16444aa3c39a07c3779454"
format_stamp: "Formatted at 2026-08-06 10:53:44 on dist-test-slave-49h3"
server_key: "a4d3ac0445968a9e323fbb690a98b73b"
server_key_iv: "982bec2d25b441efe92b5d70c4afc507"
server_key_version: "encryptionkey@0"
I20260806 10:53:44.674364 2910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:44.696656 2910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:44.702302 2910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:44.702689 2910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:44.703961 2910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:53:44.705760 2910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:44.705855 2910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:44.706001 2910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:44.706073 2910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:44.741317 2910 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:33931
I20260806 10:53:44.741397 3037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:33931 every 8 connection(s)
I20260806 10:53:44.742902 2910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:53:44.744997 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 2910
I20260806 10:53:44.745206 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:53:44.746209 31592 external_mini_cluster.cc:1428] Setting key 8ef9862e6fbca0b41815914320b29d11
Aug 06 10:53:44 dist-test-slave-49h3 krb5kdc[2804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013624, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:44.767329 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:44.746270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:40085 (local address 127.30.218.62:41915)
0806 10:53:44.746900 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:44.746906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:44.748169 (+ 1263us) server_negotiation.cc:407] Connection header received
0806 10:53:44.750414 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:44.750426 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:44.750560 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:44.750727 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:44.752519 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:44.753687 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:44.755464 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:44.756146 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:44.756312 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:44.760275 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:44.760283 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:44.760288 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:44.760341 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:44.763437 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:44.764460 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:44.764465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:44.764468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:44.764623 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:44.765108 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:44.765112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:44.765114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:44.765483 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:44.765626 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:44.766686 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0806 10:53:44.766947 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":304,"threads_started":1}
I20260806 10:53:44.768822 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:44.747023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:41915 (local address 127.30.218.1:40085)
0806 10:53:44.747966 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:44.748020 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:44.750089 (+ 2069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:44.750971 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:44.750981 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:44.751607 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:44.752297 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:44.752327 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:44.753885 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:44.753889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:44.755272 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:44.755287 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:44.756429 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:44.756434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:44.756674 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:44.757481 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:44.757523 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:44.759990 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:44.763600 (+ 3610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:44.763611 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:44.763632 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:44.764303 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:44.764741 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:44.764746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:44.764749 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:44.765018 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:44.765626 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:44.765635 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:44.766162 (+ 527us) client_negotiation.cc:769] Sending connection context
0806 10:53:44.766539 (+ 377us) client_negotiation.cc:240] Negotiation successful
0806 10:53:44.767203 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":382,"threads_started":1}
I20260806 10:53:44.770896 3038 heartbeater.cc:347] Connected to a master server at 127.30.218.62:41915
I20260806 10:53:44.771265 3038 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:44.772235 3038 heartbeater.cc:511] Master 127.30.218.62:41915 requested a full tablet report, sending...
I20260806 10:53:44.774927 2849 ts_manager.cc:194] Registered new tserver with Master: 507915a50f16444aa3c39a07c3779454 (127.30.218.1:33931)
WARNING: no policy specified for kudu/127.30.218.2@KRBTEST.COM; defaulting to no policy
I20260806 10:53:44.776726 2849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.1@KRBTEST.COM'} at 127.30.218.1:40085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
I20260806 10:53:44.812732 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:41915
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:37775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:45.197613 3044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:45.197902 3044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:45.197994 3044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:45.205305 3044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:45.205425 3044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:45.205581 3044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:53:45.215847 3044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:37775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:41915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:45.217146 3044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:45.218499 3044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:45.231957 3050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:45.231921 3052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:45.232025 3049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:45.232875 3044 server_base.cc:1034] running on GCE node
I20260806 10:53:45.233531 3044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:45.234480 3044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:45.235692 3044 hybrid_clock.cc:648] HybridClock initialized: now 1786013625235630 us; error 54 us; skew 500 ppm
I20260806 10:53:45.236006 3044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:45 dist-test-slave-49h3 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013625, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:45.239326 3044 init.cc:377] Logged in from keytab as kudu/127.30.218.2@KRBTEST.COM (short username kudu)
I20260806 10:53:45.240854 3044 webserver.cc:460] Webserver started at http://127.30.218.2:36699/ using document root <none> and password file <none>
I20260806 10:53:45.241626 3044 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:45.241755 3044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:45.242070 3044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:45.245286 3044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "9af1698c657343da83d4994dc89f1c19"
format_stamp: "Formatted at 2026-08-06 10:53:45 on dist-test-slave-49h3"
server_key: "5416f2be6ca6e45ce4a5e93f16aec3d3"
server_key_iv: "9cff3f7591f6d4612a9749d79591923b"
server_key_version: "encryptionkey@0"
I20260806 10:53:45.246238 3044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "9af1698c657343da83d4994dc89f1c19"
format_stamp: "Formatted at 2026-08-06 10:53:45 on dist-test-slave-49h3"
server_key: "5416f2be6ca6e45ce4a5e93f16aec3d3"
server_key_iv: "9cff3f7591f6d4612a9749d79591923b"
server_key_version: "encryptionkey@0"
I20260806 10:53:45.251969 3044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 10:53:45.255896 3059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:45.257385 3044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:53:45.257620 3044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "9af1698c657343da83d4994dc89f1c19"
format_stamp: "Formatted at 2026-08-06 10:53:45 on dist-test-slave-49h3"
server_key: "5416f2be6ca6e45ce4a5e93f16aec3d3"
server_key_iv: "9cff3f7591f6d4612a9749d79591923b"
server_key_version: "encryptionkey@0"
I20260806 10:53:45.257833 3044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:45.278354 3044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:45.283102 3044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:45.283412 3044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:45.284426 3044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:53:45.285948 3044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:45.286051 3044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:45.286190 3044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:45.286243 3044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:45.318677 3044 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:36819
I20260806 10:53:45.318730 3171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:36819 every 8 connection(s)
I20260806 10:53:45.320041 3044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:53:45.324455 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 3044
I20260806 10:53:45.324674 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:53:45.325248 31592 external_mini_cluster.cc:1428] Setting key 7e3cd894468cce76ce8fc3153c84e9f9
Aug 06 10:53:45 dist-test-slave-49h3 krb5kdc[2804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013625, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:45.342619 3174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:45.322805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:59951 (local address 127.30.218.62:41915)
0806 10:53:45.323288 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:45.323293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:45.324338 (+ 1045us) server_negotiation.cc:407] Connection header received
0806 10:53:45.326742 (+ 2404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:45.326749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:45.326845 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:45.326955 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:45.328812 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:45.329643 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:45.331363 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:45.331831 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:45.331954 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:45.336102 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:45.336108 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:45.336112 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:45.336154 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:45.339053 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:45.340042 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:45.340046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:45.340048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:45.340190 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:45.340700 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:45.340703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:45.340704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:45.340998 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:45.341129 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:45.342107 (+ 978us) server_negotiation.cc:299] Negotiation successful
0806 10:53:45.342319 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":358,"thread_start_us":245,"threads_started":1}
I20260806 10:53:45.344099 3175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:45.323319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:41915 (local address 127.30.218.2:59951)
0806 10:53:45.324122 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:45.324173 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:45.326385 (+ 2212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:45.327262 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:45.327273 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:45.327934 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:45.328615 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:45.328637 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:45.329823 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:45.329828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:45.331185 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:45.331203 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:45.332110 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:45.332119 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:45.332366 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:45.333169 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:45.333203 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:45.335796 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:45.339248 (+ 3452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:45.339259 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:45.339278 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:45.339913 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:45.340317 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:45.340322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:45.340325 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:45.340593 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:45.341128 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:45.341138 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:45.341658 (+ 520us) client_negotiation.cc:769] Sending connection context
0806 10:53:45.341999 (+ 341us) client_negotiation.cc:240] Negotiation successful
0806 10:53:45.342653 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":323,"threads_started":1}
I20260806 10:53:45.345759 3172 heartbeater.cc:347] Connected to a master server at 127.30.218.62:41915
I20260806 10:53:45.346103 3172 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:45.346892 3172 heartbeater.cc:511] Master 127.30.218.62:41915 requested a full tablet report, sending...
I20260806 10:53:45.348563 2849 ts_manager.cc:194] Registered new tserver with Master: 9af1698c657343da83d4994dc89f1c19 (127.30.218.2:36819)
I20260806 10:53:45.349833 2849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.2@KRBTEST.COM'} at 127.30.218.2:59951
WARNING: no policy specified for kudu/127.30.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
I20260806 10:53:45.380663 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:41915
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:37775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:45.727557 3178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:45.727883 3178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:45.728011 3178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:45.735631 3178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:45.735765 3178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:45.735921 3178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:53:45.745975 3178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:37775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:41915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:45.747198 3178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:45.748447 3178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:45.760670 3186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:45.760736 3183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:45.760713 3184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:45.761616 3178 server_base.cc:1034] running on GCE node
I20260806 10:53:45.762185 3178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:45.763134 3178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:45.764348 3178 hybrid_clock.cc:648] HybridClock initialized: now 1786013625764296 us; error 36 us; skew 500 ppm
I20260806 10:53:45.764664 3178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:45 dist-test-slave-49h3 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013625, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:45.767907 3178 init.cc:377] Logged in from keytab as kudu/127.30.218.3@KRBTEST.COM (short username kudu)
I20260806 10:53:45.769424 3178 webserver.cc:460] Webserver started at http://127.30.218.3:42337/ using document root <none> and password file <none>
I20260806 10:53:45.770167 3178 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:45.770294 3178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:45.770630 3178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:45.773972 3178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "55d6d3a699cf43e4a953ac352f78ce56"
format_stamp: "Formatted at 2026-08-06 10:53:45 on dist-test-slave-49h3"
server_key: "00ac2fdf07caafa359803a4e0f70bf9e"
server_key_iv: "508ec45babb7ad233841bd7880301d53"
server_key_version: "encryptionkey@0"
I20260806 10:53:45.774869 3178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "55d6d3a699cf43e4a953ac352f78ce56"
format_stamp: "Formatted at 2026-08-06 10:53:45 on dist-test-slave-49h3"
server_key: "00ac2fdf07caafa359803a4e0f70bf9e"
server_key_iv: "508ec45babb7ad233841bd7880301d53"
server_key_version: "encryptionkey@0"
I20260806 10:53:45.780464 3038 heartbeater.cc:503] Master 127.30.218.62:41915 was elected leader, sending a full tablet report...
I20260806 10:53:45.780994 3178 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260806 10:53:45.785615 3193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:45.787243 3178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:53:45.787451 3178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "55d6d3a699cf43e4a953ac352f78ce56"
format_stamp: "Formatted at 2026-08-06 10:53:45 on dist-test-slave-49h3"
server_key: "00ac2fdf07caafa359803a4e0f70bf9e"
server_key_iv: "508ec45babb7ad233841bd7880301d53"
server_key_version: "encryptionkey@0"
I20260806 10:53:45.787709 3178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:45.809393 3178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:45.813853 3178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:45.814144 3178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:45.815182 3178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:53:45.816648 3178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:45.816736 3178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:45.816856 3178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:45.816908 3178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:45.848834 3178 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:40997
I20260806 10:53:45.848886 3307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:40997 every 8 connection(s)
I20260806 10:53:45.850093 3178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:53:45.855320 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 3178
I20260806 10:53:45.855472 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:53:45.855886 31592 external_mini_cluster.cc:1428] Setting key 2a8605f52de0858973aa1064255a95b4
Aug 06 10:53:45 dist-test-slave-49h3 krb5kdc[2804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013625, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:45.870608 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:45.852485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:54865 (local address 127.30.218.62:41915)
0806 10:53:45.853001 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:45.853007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:45.853828 (+ 821us) server_negotiation.cc:407] Connection header received
0806 10:53:45.855641 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:45.855647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:45.855728 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:45.855827 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:45.857478 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:45.858394 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:45.860317 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:45.860786 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:45.860902 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:45.864910 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:45.864916 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:45.864919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:45.864956 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:45.867431 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:45.868191 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:45.868194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:45.868197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:45.868312 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:45.868673 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:45.868676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:45.868677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:45.868978 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:45.869101 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:45.869979 (+ 878us) server_negotiation.cc:299] Negotiation successful
0806 10:53:45.870223 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":232,"threads_started":1}
I20260806 10:53:45.871793 3311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:45.852959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:41915 (local address 127.30.218.3:54865)
0806 10:53:45.853686 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:45.853722 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:45.855356 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:45.856061 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:45.856077 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:45.856682 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:45.857295 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:45.857316 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:45.858547 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:45.858551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:45.860161 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:45.860177 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:45.861014 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:45.861022 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:45.861334 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:45.862169 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:45.862206 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:45.864664 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:45.867590 (+ 2926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:45.867599 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:45.867615 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:45.868084 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:45.868390 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:45.868393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:45.868395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:45.868584 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:45.869111 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:45.869136 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:45.869610 (+ 474us) client_negotiation.cc:769] Sending connection context
0806 10:53:45.869883 (+ 273us) client_negotiation.cc:240] Negotiation successful
0806 10:53:45.870419 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":252,"threads_started":1}
I20260806 10:53:45.873163 3308 heartbeater.cc:347] Connected to a master server at 127.30.218.62:41915
I20260806 10:53:45.873490 3308 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:45.874254 3308 heartbeater.cc:511] Master 127.30.218.62:41915 requested a full tablet report, sending...
I20260806 10:53:45.875831 2849 ts_manager.cc:194] Registered new tserver with Master: 55d6d3a699cf43e4a953ac352f78ce56 (127.30.218.3:40997)
I20260806 10:53:45.876703 2849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.3@KRBTEST.COM'} at 127.30.218.3:54865
I20260806 10:53:45.884125 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 10:53:45 dist-test-slave-49h3 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 10:53:45 dist-test-slave-49h3 krb5kdc[2804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.1@KRBTEST.COM
I20260806 10:53:45.944563 3316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:45.926422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55090 (local address 127.30.218.1:33931)
0806 10:53:45.927083 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:45.927091 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:45.927119 (+ 28us) server_negotiation.cc:407] Connection header received
0806 10:53:45.927184 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:45.927191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:45.927417 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:45.927554 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:45.929044 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:45.930195 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:45.931531 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:45.932142 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:45.932246 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:45.936909 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:45.936930 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:45.936951 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:45.937004 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:45.940860 (+ 3856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:45.941944 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:45.941954 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:45.941956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:45.942107 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:45.942641 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:45.942646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:45.942649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:45.943128 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:45.943439 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:45.943945 (+ 506us) server_negotiation.cc:299] Negotiation successful
0806 10:53:45.944244 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":313,"threads_started":1}
Aug 06 10:53:45 dist-test-slave-49h3 krb5kdc[2804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:45.963574 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:45.950117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41592 (local address 127.30.218.62:41915)
0806 10:53:45.950342 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:45.950347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:45.950381 (+ 34us) server_negotiation.cc:407] Connection header received
0806 10:53:45.950537 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:45.950541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:45.950607 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:45.950687 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:45.952000 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:45.952748 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:45.954042 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:45.954483 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:45.954586 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:45.958611 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:45.958616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:45.958619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:45.958659 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:45.961076 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:45.961751 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:45.961754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:45.961756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:45.961893 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:45.962239 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:45.962243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:45.962246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:45.962606 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:45.962798 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:45.963085 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 10:53:45.963270 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260806 10:53:45.964267 2849 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41592
I20260806 10:53:45.965318 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 2910
I20260806 10:53:45.978524 3316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:45.966203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55098 (local address 127.30.218.1:33931)
0806 10:53:45.966380 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:45.966386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:45.966693 (+ 307us) server_negotiation.cc:407] Connection header received
0806 10:53:45.966870 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:45.966874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:45.966968 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:45.967050 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:45.968300 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:45.969005 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:45.970347 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:45.970771 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:45.970889 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:45.973256 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:45.973261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:45.973264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:45.973299 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:45.975836 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:45.976552 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:45.976556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:45.976559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:45.976700 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:45.977092 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:45.977096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:45.977098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:45.977383 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:45.977648 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:45.977956 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 10:53:45.978217 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 10:53:45.979149 3033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:46.063275 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 2910
I20260806 10:53:46.080250 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 3044
Aug 06 10:53:46 dist-test-slave-49h3 krb5kdc[2804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.2@KRBTEST.COM
I20260806 10:53:46.097826 3321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:46.081202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60690 (local address 127.30.218.2:36819)
0806 10:53:46.081788 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:46.081797 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:46.081826 (+ 29us) server_negotiation.cc:407] Connection header received
0806 10:53:46.081903 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:46.081910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:46.082124 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:46.082261 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:46.083511 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:46.084598 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:46.085976 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:46.086613 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:46.086717 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:46.091169 (+ 4452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:46.091186 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:46.091203 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:46.091260 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:46.094613 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:46.095460 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:46.095469 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:46.095472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:46.095634 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:46.096037 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:46.096041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:46.096043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:46.096471 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:46.096784 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:46.097136 (+ 352us) server_negotiation.cc:299] Negotiation successful
0806 10:53:46.097473 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":357,"threads_started":1}
I20260806 10:53:46.099208 3167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:46.183382 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 3044
I20260806 10:53:46.200911 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 3178
Aug 06 10:53:46 dist-test-slave-49h3 krb5kdc[2804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.3@KRBTEST.COM
I20260806 10:53:46.218739 3323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:46.202123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59422 (local address 127.30.218.3:40997)
0806 10:53:46.202870 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:46.202877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:46.202909 (+ 32us) server_negotiation.cc:407] Connection header received
0806 10:53:46.202972 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:46.202978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:46.203232 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:46.203367 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:46.204653 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:46.205864 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:46.207307 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:46.207938 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:46.208046 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:46.212636 (+ 4590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:46.212653 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:46.212670 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:46.212720 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:46.215792 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:46.216652 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:46.216660 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:46.216662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:46.216822 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:46.217143 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:46.217146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:46.217148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:46.217589 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:46.217849 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:46.218126 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 10:53:46.218371 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":404,"threads_started":1}
I20260806 10:53:46.220088 3303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:46.304342 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 3178
I20260806 10:53:46.321122 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 2817
I20260806 10:53:46.322489 2879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:46.430119 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 2817
2026-08-06T10:53:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4153 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:53:46 dist-test-slave-49h3 krb5kdc[3328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:53:46 dist-test-slave-49h3 krb5kdc[3328](info): set up 2 sockets
Aug 06 10:53:46 dist-test-slave-49h3 krb5kdc[3328](info): commencing operation
krb5kdc: starting...
W20260806 10:53:47.478792 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.004s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:53:47 dist-test-slave-49h3 krb5kdc[3328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:53:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:53:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:53:47.732479 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:34085
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:34085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:48.074774 3342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:48.075120 3342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:48.075253 3342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:48.082973 3342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:53:48.083129 3342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:48.083206 3342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:48.083293 3342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:53:48.083357 3342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:53:48.094149 3342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:34085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:34085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:48.095620 3342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:48.097023 3342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:48.109326 3348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:48.109298 3347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:48.109543 3350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:48.110172 3342 server_base.cc:1034] running on GCE node
I20260806 10:53:48.111054 3342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:48.112643 3342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:48.113855 3342 hybrid_clock.cc:648] HybridClock initialized: now 1786013628113801 us; error 52 us; skew 500 ppm
I20260806 10:53:48.114228 3342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:48 dist-test-slave-49h3 krb5kdc[3328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013628, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:48.117688 3342 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:53:48.118935 3342 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:48.119047 3342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:48.119385 3342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:48.122608 3342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "97603a5a930c46309c61b9a6b2d31428"
format_stamp: "Formatted at 2026-08-06 10:53:48 on dist-test-slave-49h3"
server_key: "58ddd7ca16efa114d82da1245c900905"
server_key_iv: "3c248ee67b1644a8c4a396c9785d2026"
server_key_version: "encryptionkey@0"
I20260806 10:53:48.123442 3342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "97603a5a930c46309c61b9a6b2d31428"
format_stamp: "Formatted at 2026-08-06 10:53:48 on dist-test-slave-49h3"
server_key: "58ddd7ca16efa114d82da1245c900905"
server_key_iv: "3c248ee67b1644a8c4a396c9785d2026"
server_key_version: "encryptionkey@0"
I20260806 10:53:48.128834 3342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260806 10:53:48.132527 3356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:48.134061 3342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:53:48.134244 3342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "97603a5a930c46309c61b9a6b2d31428"
format_stamp: "Formatted at 2026-08-06 10:53:48 on dist-test-slave-49h3"
server_key: "58ddd7ca16efa114d82da1245c900905"
server_key_iv: "3c248ee67b1644a8c4a396c9785d2026"
server_key_version: "encryptionkey@0"
I20260806 10:53:48.134469 3342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:48.153906 3342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:48.158139 3342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:48.158401 3342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:48.177040 3342 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:34085
I20260806 10:53:48.177013 3407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:34085 every 8 connection(s)
I20260806 10:53:48.178107 3342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:53:48.182194 3408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:48.187901 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 3342
I20260806 10:53:48.188115 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:53:48.188911 31592 external_mini_cluster.cc:1428] Setting key 72f7fde03cc58b3ef2078b0e76ba232f
I20260806 10:53:48.194305 3408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97603a5a930c46309c61b9a6b2d31428: Bootstrap starting.
I20260806 10:53:48.197543 3408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97603a5a930c46309c61b9a6b2d31428: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:48.198642 3408 log.cc:824] T 00000000000000000000000000000000 P 97603a5a930c46309c61b9a6b2d31428: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:48.201958 3408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97603a5a930c46309c61b9a6b2d31428: No bootstrap required, opened a new log
Aug 06 10:53:48 dist-test-slave-49h3 krb5kdc[3328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:48.211283 3408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97603a5a930c46309c61b9a6b2d31428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97603a5a930c46309c61b9a6b2d31428" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34085 } }
I20260806 10:53:48.211798 3408 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97603a5a930c46309c61b9a6b2d31428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:48.211928 3408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97603a5a930c46309c61b9a6b2d31428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97603a5a930c46309c61b9a6b2d31428, State: Initialized, Role: FOLLOWER
I20260806 10:53:48.212586 3408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97603a5a930c46309c61b9a6b2d31428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97603a5a930c46309c61b9a6b2d31428" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34085 } }
I20260806 10:53:48.212837 3408 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97603a5a930c46309c61b9a6b2d31428 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:53:48.212844 3411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:48.191222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54536 (local address 127.30.218.62:34085)
0806 10:53:48.192043 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:48.192060 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:48.192107 (+ 47us) server_negotiation.cc:407] Connection header received
0806 10:53:48.193904 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:48.193932 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:48.194337 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:48.194736 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:48.196293 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:48.197686 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:48.199007 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:48.200097 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:48.200271 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:48.204933 (+ 4662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:48.204942 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:48.204963 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:48.205006 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:48.208322 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:48.209292 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:48.209300 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:48.209308 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:48.209580 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:48.210000 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:48.210003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:48.210005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:48.210640 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:48.210851 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:48.211373 (+ 522us) server_negotiation.cc:299] Negotiation successful
0806 10:53:48.211723 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":344,"threads_started":1}
I20260806 10:53:48.212990 3408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97603a5a930c46309c61b9a6b2d31428 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:53:48.213238 3408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97603a5a930c46309c61b9a6b2d31428 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:48.216044 3408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97603a5a930c46309c61b9a6b2d31428 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97603a5a930c46309c61b9a6b2d31428" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34085 } }
I20260806 10:53:48.216526 3408 leader_election.cc:302] T 00000000000000000000000000000000 P 97603a5a930c46309c61b9a6b2d31428 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97603a5a930c46309c61b9a6b2d31428; no voters:
I20260806 10:53:48.217053 3408 leader_election.cc:288] T 00000000000000000000000000000000 P 97603a5a930c46309c61b9a6b2d31428 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:53:48.217175 3413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97603a5a930c46309c61b9a6b2d31428 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:48.217909 3413 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97603a5a930c46309c61b9a6b2d31428 [term 1 LEADER]: Becoming Leader. State: Replica: 97603a5a930c46309c61b9a6b2d31428, State: Running, Role: LEADER
I20260806 10:53:48.218430 3413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97603a5a930c46309c61b9a6b2d31428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97603a5a930c46309c61b9a6b2d31428" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34085 } }
I20260806 10:53:48.219125 3408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97603a5a930c46309c61b9a6b2d31428 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:53:48.221050 3415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97603a5a930c46309c61b9a6b2d31428 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97603a5a930c46309c61b9a6b2d31428. Latest consensus state: current_term: 1 leader_uuid: "97603a5a930c46309c61b9a6b2d31428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97603a5a930c46309c61b9a6b2d31428" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34085 } } }
I20260806 10:53:48.221122 3414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97603a5a930c46309c61b9a6b2d31428 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97603a5a930c46309c61b9a6b2d31428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97603a5a930c46309c61b9a6b2d31428" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34085 } } }
I20260806 10:53:48.221397 3415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97603a5a930c46309c61b9a6b2d31428 [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:48.221486 3414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97603a5a930c46309c61b9a6b2d31428 [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:48.222325 3420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:53:48.228327 3420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:53:48.237902 3420 catalog_manager.cc:1347] Generated new cluster ID: e9bb55d154454ad49c80410a6bb76e74
I20260806 10:53:48.238030 3420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:53:48.270007 3420 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:53:48.272034 3420 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:53:48.283927 3420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97603a5a930c46309c61b9a6b2d31428: Generated new TSK 0
I20260806 10:53:48.284876 3420 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
I20260806 10:53:48.361130 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:34085
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:48.733023 3434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:48.733374 3434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:48.733553 3434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:48.741328 3434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:48.741498 3434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:48.741660 3434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:53:48.752105 3434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:34085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:48.753571 3434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:48.755029 3434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:48.768332 3442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:48.768370 3440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:48.768690 3439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:48.769245 3434 server_base.cc:1034] running on GCE node
I20260806 10:53:48.769955 3434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:48.770983 3434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:48.772223 3434 hybrid_clock.cc:648] HybridClock initialized: now 1786013628772168 us; error 49 us; skew 500 ppm
I20260806 10:53:48.772535 3434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:48 dist-test-slave-49h3 krb5kdc[3328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013628, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:48.775707 3434 init.cc:377] Logged in from keytab as kudu/127.30.218.1@KRBTEST.COM (short username kudu)
I20260806 10:53:48.776957 3434 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:48.777063 3434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:48.777372 3434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:48.780435 3434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "0b8e3ef7313048ccb808f078eea9607c"
format_stamp: "Formatted at 2026-08-06 10:53:48 on dist-test-slave-49h3"
server_key: "12aa0348e38eed028f84c7edf013fb19"
server_key_iv: "4ecb32aa882dd4631814501f589d9a1b"
server_key_version: "encryptionkey@0"
I20260806 10:53:48.781322 3434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "0b8e3ef7313048ccb808f078eea9607c"
format_stamp: "Formatted at 2026-08-06 10:53:48 on dist-test-slave-49h3"
server_key: "12aa0348e38eed028f84c7edf013fb19"
server_key_iv: "4ecb32aa882dd4631814501f589d9a1b"
server_key_version: "encryptionkey@0"
I20260806 10:53:48.786727 3434 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 10:53:48.790537 3448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:48.791997 3434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:53:48.792184 3434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "0b8e3ef7313048ccb808f078eea9607c"
format_stamp: "Formatted at 2026-08-06 10:53:48 on dist-test-slave-49h3"
server_key: "12aa0348e38eed028f84c7edf013fb19"
server_key_iv: "4ecb32aa882dd4631814501f589d9a1b"
server_key_version: "encryptionkey@0"
I20260806 10:53:48.792429 3434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:48.805583 3434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:48.809763 3434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:48.810022 3434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:48.810988 3434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:53:48.812373 3434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:48.812470 3434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:48.812585 3434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:48.812657 3434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:48.845638 3434 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:34453
I20260806 10:53:48.845695 3561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:34453 every 8 connection(s)
I20260806 10:53:48.846563 3434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:53:48.852520 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 3434
I20260806 10:53:48.852727 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:53:48.853364 31592 external_mini_cluster.cc:1428] Setting key 38802962c9a4c728a5aeedc7da39d133
Aug 06 10:53:48 dist-test-slave-49h3 krb5kdc[3328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013628, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:48.866434 3564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:48.849108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:37831 (local address 127.30.218.62:34085)
0806 10:53:48.849648 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:48.849653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:48.850597 (+ 944us) server_negotiation.cc:407] Connection header received
0806 10:53:48.852583 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:48.852589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:48.852679 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:48.852803 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:48.854570 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:48.855426 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:48.857042 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:48.857437 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:48.857596 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:48.861095 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:48.861099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:48.861103 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:48.861133 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:48.863424 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:48.864124 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:48.864128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:48.864131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:48.864253 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:48.864660 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:48.864665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:48.864668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:48.864937 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:48.865083 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:48.865820 (+ 737us) server_negotiation.cc:299] Negotiation successful
0806 10:53:48.866077 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":248,"threads_started":1}
I20260806 10:53:48.867458 3565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:48.849701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34085 (local address 127.30.218.1:37831)
0806 10:53:48.850427 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:48.850477 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:48.852282 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:48.853071 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:48.853081 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:48.853705 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:48.854387 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:48.854413 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:48.855590 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:48.855594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:48.856889 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:48.856903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:48.857692 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:48.857700 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:48.857928 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:48.858575 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:48.858598 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:48.860880 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:48.863557 (+ 2677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:48.863566 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:48.863583 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:48.864001 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:48.864355 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:48.864360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:48.864364 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:48.864552 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:48.865069 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:48.865076 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:48.865413 (+ 337us) client_negotiation.cc:769] Sending connection context
0806 10:53:48.865704 (+ 291us) client_negotiation.cc:240] Negotiation successful
0806 10:53:48.866257 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":283,"threads_started":1}
I20260806 10:53:48.869166 3562 heartbeater.cc:347] Connected to a master server at 127.30.218.62:34085
I20260806 10:53:48.869513 3562 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:48.870234 3562 heartbeater.cc:511] Master 127.30.218.62:34085 requested a full tablet report, sending...
I20260806 10:53:48.873134 3373 ts_manager.cc:194] Registered new tserver with Master: 0b8e3ef7313048ccb808f078eea9607c (127.30.218.1:34453)
I20260806 10:53:48.875373 3373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.1@KRBTEST.COM'} at 127.30.218.1:37831
WARNING: no policy specified for kudu/127.30.218.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
I20260806 10:53:48.908798 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:34085
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:49.259200 3568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:49.259500 3568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:49.259635 3568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:49.267045 3568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:49.267163 3568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:49.267314 3568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:53:49.277330 3568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:34085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:49.278779 3568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:49.280125 3568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:49.292934 3573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:49.292886 3576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:49.292866 3574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:49.293721 3568 server_base.cc:1034] running on GCE node
I20260806 10:53:49.294277 3568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:49.295192 3568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:49.296389 3568 hybrid_clock.cc:648] HybridClock initialized: now 1786013629296351 us; error 31 us; skew 500 ppm
I20260806 10:53:49.296707 3568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:49 dist-test-slave-49h3 krb5kdc[3328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013629, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:49.304173 3568 init.cc:377] Logged in from keytab as kudu/127.30.218.2@KRBTEST.COM (short username kudu)
I20260806 10:53:49.305330 3568 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:49.305436 3568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:49.305809 3568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:49.308736 3568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "472a937cc01f4c688603af99d1798571"
format_stamp: "Formatted at 2026-08-06 10:53:49 on dist-test-slave-49h3"
server_key: "8d8403a2050a3c510dd45fabc5654434"
server_key_iv: "6ec9e0a2b38f86d2b8855934c2dc8f21"
server_key_version: "encryptionkey@0"
I20260806 10:53:49.309623 3568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "472a937cc01f4c688603af99d1798571"
format_stamp: "Formatted at 2026-08-06 10:53:49 on dist-test-slave-49h3"
server_key: "8d8403a2050a3c510dd45fabc5654434"
server_key_iv: "6ec9e0a2b38f86d2b8855934c2dc8f21"
server_key_version: "encryptionkey@0"
I20260806 10:53:49.314873 3568 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 10:53:49.318589 3582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:49.320035 3568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:53:49.320222 3568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "472a937cc01f4c688603af99d1798571"
format_stamp: "Formatted at 2026-08-06 10:53:49 on dist-test-slave-49h3"
server_key: "8d8403a2050a3c510dd45fabc5654434"
server_key_iv: "6ec9e0a2b38f86d2b8855934c2dc8f21"
server_key_version: "encryptionkey@0"
I20260806 10:53:49.320446 3568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:49.357402 3568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:49.361959 3568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:49.362259 3568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:49.363354 3568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:53:49.364864 3568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:49.364975 3568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:49.365096 3568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:49.365170 3568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:49.398039 3568 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:41185
I20260806 10:53:49.398104 3695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:41185 every 8 connection(s)
I20260806 10:53:49.398960 3568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:53:49.405251 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 3568
I20260806 10:53:49.405478 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:53:49.406132 31592 external_mini_cluster.cc:1428] Setting key a7ae29882f20167b27fe7581ef4f6e1e
Aug 06 10:53:49 dist-test-slave-49h3 krb5kdc[3328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013629, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:49.420588 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:49.401547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:37371 (local address 127.30.218.62:34085)
0806 10:53:49.402055 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:49.402061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:49.403163 (+ 1102us) server_negotiation.cc:407] Connection header received
0806 10:53:49.405007 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:49.405012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:49.405093 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:49.405201 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:49.407219 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:49.407955 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:49.410173 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:49.410622 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:49.410729 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:49.414815 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:49.414821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:49.414824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:49.414859 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:49.417514 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:49.418212 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:49.418218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:49.418222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:49.418381 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:49.418754 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:49.418758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:49.418761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:49.419099 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:49.419258 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:49.420008 (+ 750us) server_negotiation.cc:299] Negotiation successful
0806 10:53:49.420232 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":214,"threads_started":1}
I20260806 10:53:49.421437 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:49.402168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34085 (local address 127.30.218.2:37371)
0806 10:53:49.402961 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:49.403003 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:49.404697 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:49.405501 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:49.405514 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:49.406297 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:49.407018 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:49.407046 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:49.408120 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:49.408125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:49.409976 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:49.409993 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:49.410851 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:49.410858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:49.411129 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:49.411861 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:49.411889 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:49.414557 (+ 2668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:49.417640 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:49.417647 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:49.417662 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:49.418086 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:49.418470 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:49.418473 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:49.418475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:49.418644 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:49.419297 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:49.419307 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:49.419630 (+ 323us) client_negotiation.cc:769] Sending connection context
0806 10:53:49.419894 (+ 264us) client_negotiation.cc:240] Negotiation successful
0806 10:53:49.420300 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":337,"threads_started":1}
I20260806 10:53:49.422860 3696 heartbeater.cc:347] Connected to a master server at 127.30.218.62:34085
I20260806 10:53:49.423149 3696 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:49.423763 3696 heartbeater.cc:511] Master 127.30.218.62:34085 requested a full tablet report, sending...
I20260806 10:53:49.425168 3373 ts_manager.cc:194] Registered new tserver with Master: 472a937cc01f4c688603af99d1798571 (127.30.218.2:41185)
I20260806 10:53:49.426079 3373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.2@KRBTEST.COM'} at 127.30.218.2:37371
WARNING: no policy specified for kudu/127.30.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
I20260806 10:53:49.460785 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:34085
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:49.793923 3702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:49.794224 3702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:49.794374 3702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:49.801687 3702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:49.801815 3702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:49.801967 3702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:53:49.812078 3702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:34085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:49.813552 3702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:49.814994 3702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:49.828338 3707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:49.828270 3710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:49.828296 3708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:49.829177 3702 server_base.cc:1034] running on GCE node
I20260806 10:53:49.829833 3702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:49.830775 3702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:49.831959 3702 hybrid_clock.cc:648] HybridClock initialized: now 1786013629831898 us; error 54 us; skew 500 ppm
I20260806 10:53:49.832265 3702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:49 dist-test-slave-49h3 krb5kdc[3328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013629, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:49.835224 3702 init.cc:377] Logged in from keytab as kudu/127.30.218.3@KRBTEST.COM (short username kudu)
I20260806 10:53:49.836310 3702 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:49.836400 3702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:49.836653 3702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:49.839650 3702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "8d60f4b180214729a0b6402a8743b234"
format_stamp: "Formatted at 2026-08-06 10:53:49 on dist-test-slave-49h3"
server_key: "e5a2e6d12ad83544e731730a0d39453d"
server_key_iv: "fe48f0f3034abfa92afda46c0d713c55"
server_key_version: "encryptionkey@0"
I20260806 10:53:49.840440 3702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "8d60f4b180214729a0b6402a8743b234"
format_stamp: "Formatted at 2026-08-06 10:53:49 on dist-test-slave-49h3"
server_key: "e5a2e6d12ad83544e731730a0d39453d"
server_key_iv: "fe48f0f3034abfa92afda46c0d713c55"
server_key_version: "encryptionkey@0"
I20260806 10:53:49.845676 3702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260806 10:53:49.850194 3716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:49.851549 3702 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 10:53:49.851716 3702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "8d60f4b180214729a0b6402a8743b234"
format_stamp: "Formatted at 2026-08-06 10:53:49 on dist-test-slave-49h3"
server_key: "e5a2e6d12ad83544e731730a0d39453d"
server_key_iv: "fe48f0f3034abfa92afda46c0d713c55"
server_key_version: "encryptionkey@0"
I20260806 10:53:49.851933 3702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:49.878013 3702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:49.879143 3562 heartbeater.cc:503] Master 127.30.218.62:34085 was elected leader, sending a full tablet report...
I20260806 10:53:49.882808 3702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:49.883073 3702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:49.884037 3702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:53:49.885437 3702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:49.885576 3702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:49.885684 3702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:49.885752 3702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:49.915649 3702 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:46019
I20260806 10:53:49.915710 3828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:46019 every 8 connection(s)
I20260806 10:53:49.916509 3702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:53:49.926488 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 3702
I20260806 10:53:49.926647 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:53:49.927083 31592 external_mini_cluster.cc:1428] Setting key cf88ccfb00f21f6ecd1b592027136f17
Aug 06 10:53:49 dist-test-slave-49h3 krb5kdc[3328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013629, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:49.938498 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:49.919007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:54779 (local address 127.30.218.62:34085)
0806 10:53:49.919247 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:49.919254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:49.920469 (+ 1215us) server_negotiation.cc:407] Connection header received
0806 10:53:49.922595 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:49.922600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:49.922699 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:49.922788 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:49.924477 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:49.925280 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:49.927168 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:49.927632 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:49.927768 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:49.931917 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:49.931924 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:49.931927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:49.931965 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:49.934617 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:49.935502 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:49.935507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:49.935509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:49.935654 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:49.936108 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:49.936112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:49.936114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:49.936414 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:49.936574 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:49.937514 (+ 940us) server_negotiation.cc:299] Negotiation successful
0806 10:53:49.937942 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 10:53:49.939028 3831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:49.919495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34085 (local address 127.30.218.3:54779)
0806 10:53:49.920297 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:49.920345 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:49.922280 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:49.923002 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:49.923010 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:49.923590 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:49.924304 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:49.924325 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:49.925478 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:49.925485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:49.927004 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:49.927019 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:49.927896 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:49.927904 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:49.928180 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:49.928946 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:53:49.928972 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:49.931691 (+ 2719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:49.934804 (+ 3113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:49.934815 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:49.934837 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:49.935355 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:49.935769 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:53:49.935773 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:53:49.935775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:53:49.935980 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:53:49.936554 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:49.936563 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:49.936955 (+ 392us) client_negotiation.cc:769] Sending connection context
0806 10:53:49.937227 (+ 272us) client_negotiation.cc:240] Negotiation successful
0806 10:53:49.937744 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":295,"threads_started":1}
I20260806 10:53:49.940889 3829 heartbeater.cc:347] Connected to a master server at 127.30.218.62:34085
I20260806 10:53:49.941208 3829 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:49.942154 3829 heartbeater.cc:511] Master 127.30.218.62:34085 requested a full tablet report, sending...
I20260806 10:53:49.943912 3373 ts_manager.cc:194] Registered new tserver with Master: 8d60f4b180214729a0b6402a8743b234 (127.30.218.3:46019)
I20260806 10:53:49.944845 3373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.3@KRBTEST.COM'} at 127.30.218.3:54779
I20260806 10:53:49.956892 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:53:49.973721 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:49.961593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54550 (local address 127.30.218.62:34085)
0806 10:53:49.961812 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:49.961819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:49.962161 (+ 342us) server_negotiation.cc:407] Connection header received
0806 10:53:49.962294 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:49.962298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:49.962377 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:49.962459 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:49.963817 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:49.964527 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:49.965938 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:49.966402 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:49.966502 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:49.968601 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:49.968606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:49.968609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:49.968640 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:49.971217 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:49.971894 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:49.971898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:49.971901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:49.972054 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:49.972402 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:49.972406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:49.972409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:49.972747 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:49.972954 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:49.973191 (+ 237us) server_negotiation.cc:299] Negotiation successful
0806 10:53:49.973365 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 10:53:49.978031 3373 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:53:49.985280 3373 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:53:50.013278 3841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:49.998492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:46019 (local address 127.0.0.1:35118)
0806 10:53:49.999334 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:49.999380 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:49.999585 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:50.001208 (+ 1623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:50.001215 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:50.001248 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:53:50.001693 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:50.001710 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.005500 (+ 3790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.005507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:50.008301 (+ 2794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:50.008325 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.012094 (+ 3769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.012102 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:50.012365 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:50.012432 (+ 67us) client_negotiation.cc:769] Sending connection context
0806 10:53:50.012558 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 10:53:50.012793 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":351,"threads_started":1}
I20260806 10:53:50.015381 3842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:49.998696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35118 (local address 127.30.218.3:46019)
0806 10:53:50.000535 (+ 1839us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:50.000545 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:50.000580 (+ 35us) server_negotiation.cc:407] Connection header received
0806 10:53:50.000681 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:50.000688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:50.000964 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:50.001938 (+ 974us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:53:50.003622 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.005288 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.010114 (+ 4826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.011821 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.014438 (+ 2617us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:50.014771 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:50.014855 (+ 84us) server_negotiation.cc:299] Negotiation successful
0806 10:53:50.015058 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1660,"thread_start_us":1514,"threads_started":1}
I20260806 10:53:50.019425 3764 tablet_service.cc:1467] Processing CreateTablet for tablet bd43b1c835744d84870b92f07b9e92ae (DEFAULT_TABLE table=test-table [id=f3955b33a0134893be4970dd7d7e3aa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:50.021021 3764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd43b1c835744d84870b92f07b9e92ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:50.021919 3846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:50.001416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54958 (local address 127.30.218.2:41185)
0806 10:53:50.014182 (+ 12766us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:50.014191 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:50.014224 (+ 33us) server_negotiation.cc:407] Connection header received
0806 10:53:50.014328 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:50.014336 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:50.014650 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:50.014859 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:53:50.016341 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.017372 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.019635 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.020857 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.020983 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:50.021199 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:50.021412 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 10:53:50.021604 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12581,"thread_start_us":13238,"threads_started":1}
I20260806 10:53:50.021943 3845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:50.000899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:41185 (local address 127.0.0.1:54958)
0806 10:53:50.013588 (+ 12689us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:50.013614 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:50.013767 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:50.015100 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:50.015105 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:50.015133 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:53:50.015464 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:50.015477 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.017534 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.017540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:50.019439 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:50.019455 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.021053 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.021059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:50.021245 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:50.021273 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 10:53:50.021355 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 10:53:50.021517 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12513,"mutex_wait_us":1,"thread_start_us":12961,"threads_started":1}
I20260806 10:53:50.023936 3843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:49.999681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:34453 (local address 127.0.0.1:55870)
0806 10:53:50.000776 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:50.000792 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:50.000901 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:50.015939 (+ 15038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:50.015943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:50.015966 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:53:50.016307 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:50.016320 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.018177 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.018183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:50.020244 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:50.020260 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.023159 (+ 2899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.023166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:50.023335 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:50.023370 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 10:53:50.023461 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 10:53:50.023596 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"thread_start_us":851,"threads_started":1}
I20260806 10:53:50.024866 3844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:50.000207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55870 (local address 127.30.218.1:34453)
0806 10:53:50.015380 (+ 15173us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:50.015389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:50.015416 (+ 27us) server_negotiation.cc:407] Connection header received
0806 10:53:50.015524 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:50.015532 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:50.015743 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:50.015940 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:53:50.016507 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.018028 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.020414 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.022919 (+ 2505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.023058 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:50.023269 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:50.023933 (+ 664us) server_negotiation.cc:299] Negotiation successful
0806 10:53:50.024513 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15002,"thread_start_us":14829,"threads_started":1}
I20260806 10:53:50.025538 3630 tablet_service.cc:1467] Processing CreateTablet for tablet bd43b1c835744d84870b92f07b9e92ae (DEFAULT_TABLE table=test-table [id=f3955b33a0134893be4970dd7d7e3aa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:50.026892 3630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd43b1c835744d84870b92f07b9e92ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:50.029642 3496 tablet_service.cc:1467] Processing CreateTablet for tablet bd43b1c835744d84870b92f07b9e92ae (DEFAULT_TABLE table=test-table [id=f3955b33a0134893be4970dd7d7e3aa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:50.031121 3496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd43b1c835744d84870b92f07b9e92ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:50.034453 3847 tablet_bootstrap.cc:492] T bd43b1c835744d84870b92f07b9e92ae P 8d60f4b180214729a0b6402a8743b234: Bootstrap starting.
I20260806 10:53:50.037712 3847 tablet_bootstrap.cc:654] T bd43b1c835744d84870b92f07b9e92ae P 8d60f4b180214729a0b6402a8743b234: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:50.038904 3847 log.cc:824] T bd43b1c835744d84870b92f07b9e92ae P 8d60f4b180214729a0b6402a8743b234: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:50.042423 3847 tablet_bootstrap.cc:492] T bd43b1c835744d84870b92f07b9e92ae P 8d60f4b180214729a0b6402a8743b234: No bootstrap required, opened a new log
I20260806 10:53:50.042739 3847 ts_tablet_manager.cc:1397] T bd43b1c835744d84870b92f07b9e92ae P 8d60f4b180214729a0b6402a8743b234: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260806 10:53:50.043042 3849 tablet_bootstrap.cc:492] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571: Bootstrap starting.
I20260806 10:53:50.046079 3850 tablet_bootstrap.cc:492] T bd43b1c835744d84870b92f07b9e92ae P 0b8e3ef7313048ccb808f078eea9607c: Bootstrap starting.
I20260806 10:53:50.046888 3849 tablet_bootstrap.cc:654] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:50.047915 3849 log.cc:824] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:50.049721 3850 tablet_bootstrap.cc:654] T bd43b1c835744d84870b92f07b9e92ae P 0b8e3ef7313048ccb808f078eea9607c: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:50.050525 3849 tablet_bootstrap.cc:492] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571: No bootstrap required, opened a new log
I20260806 10:53:50.050784 3849 ts_tablet_manager.cc:1397] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260806 10:53:50.050892 3850 log.cc:824] T bd43b1c835744d84870b92f07b9e92ae P 0b8e3ef7313048ccb808f078eea9607c: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:50.051982 3847 raft_consensus.cc:348] T bd43b1c835744d84870b92f07b9e92ae P 8d60f4b180214729a0b6402a8743b234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d60f4b180214729a0b6402a8743b234" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 46019 } } peers { permanent_uuid: "0b8e3ef7313048ccb808f078eea9607c" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34453 } } peers { permanent_uuid: "472a937cc01f4c688603af99d1798571" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 41185 } }
I20260806 10:53:50.052374 3847 raft_consensus.cc:374] T bd43b1c835744d84870b92f07b9e92ae P 8d60f4b180214729a0b6402a8743b234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:50.052476 3847 raft_consensus.cc:729] T bd43b1c835744d84870b92f07b9e92ae P 8d60f4b180214729a0b6402a8743b234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d60f4b180214729a0b6402a8743b234, State: Initialized, Role: FOLLOWER
I20260806 10:53:50.053090 3847 consensus_queue.cc:260] T bd43b1c835744d84870b92f07b9e92ae P 8d60f4b180214729a0b6402a8743b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d60f4b180214729a0b6402a8743b234" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 46019 } } peers { permanent_uuid: "0b8e3ef7313048ccb808f078eea9607c" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34453 } } peers { permanent_uuid: "472a937cc01f4c688603af99d1798571" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 41185 } }
I20260806 10:53:50.054765 3850 tablet_bootstrap.cc:492] T bd43b1c835744d84870b92f07b9e92ae P 0b8e3ef7313048ccb808f078eea9607c: No bootstrap required, opened a new log
I20260806 10:53:50.055050 3850 ts_tablet_manager.cc:1397] T bd43b1c835744d84870b92f07b9e92ae P 0b8e3ef7313048ccb808f078eea9607c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260806 10:53:50.059034 3849 raft_consensus.cc:348] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d60f4b180214729a0b6402a8743b234" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 46019 } } peers { permanent_uuid: "0b8e3ef7313048ccb808f078eea9607c" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34453 } } peers { permanent_uuid: "472a937cc01f4c688603af99d1798571" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 41185 } }
I20260806 10:53:50.059372 3849 raft_consensus.cc:374] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:50.059464 3849 raft_consensus.cc:729] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 472a937cc01f4c688603af99d1798571, State: Initialized, Role: FOLLOWER
I20260806 10:53:50.059983 3849 consensus_queue.cc:260] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d60f4b180214729a0b6402a8743b234" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 46019 } } peers { permanent_uuid: "0b8e3ef7313048ccb808f078eea9607c" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34453 } } peers { permanent_uuid: "472a937cc01f4c688603af99d1798571" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 41185 } }
I20260806 10:53:50.060948 3696 heartbeater.cc:503] Master 127.30.218.62:34085 was elected leader, sending a full tablet report...
I20260806 10:53:50.062362 3849 ts_tablet_manager.cc:1428] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260806 10:53:50.066423 3829 heartbeater.cc:503] Master 127.30.218.62:34085 was elected leader, sending a full tablet report...
I20260806 10:53:50.066844 3847 ts_tablet_manager.cc:1428] T bd43b1c835744d84870b92f07b9e92ae P 8d60f4b180214729a0b6402a8743b234: Time spent starting tablet: real 0.024s user 0.007s sys 0.017s
I20260806 10:53:50.067162 3850 raft_consensus.cc:348] T bd43b1c835744d84870b92f07b9e92ae P 0b8e3ef7313048ccb808f078eea9607c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d60f4b180214729a0b6402a8743b234" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 46019 } } peers { permanent_uuid: "0b8e3ef7313048ccb808f078eea9607c" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34453 } } peers { permanent_uuid: "472a937cc01f4c688603af99d1798571" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 41185 } }
I20260806 10:53:50.067565 3850 raft_consensus.cc:374] T bd43b1c835744d84870b92f07b9e92ae P 0b8e3ef7313048ccb808f078eea9607c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:50.067660 3850 raft_consensus.cc:729] T bd43b1c835744d84870b92f07b9e92ae P 0b8e3ef7313048ccb808f078eea9607c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b8e3ef7313048ccb808f078eea9607c, State: Initialized, Role: FOLLOWER
I20260806 10:53:50.068212 3850 consensus_queue.cc:260] T bd43b1c835744d84870b92f07b9e92ae P 0b8e3ef7313048ccb808f078eea9607c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d60f4b180214729a0b6402a8743b234" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 46019 } } peers { permanent_uuid: "0b8e3ef7313048ccb808f078eea9607c" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34453 } } peers { permanent_uuid: "472a937cc01f4c688603af99d1798571" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 41185 } }
I20260806 10:53:50.068590 3854 raft_consensus.cc:482] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:53:50.069006 3854 raft_consensus.cc:504] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d60f4b180214729a0b6402a8743b234" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 46019 } } peers { permanent_uuid: "0b8e3ef7313048ccb808f078eea9607c" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34453 } } peers { permanent_uuid: "472a937cc01f4c688603af99d1798571" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 41185 } }
I20260806 10:53:50.069836 3850 ts_tablet_manager.cc:1428] T bd43b1c835744d84870b92f07b9e92ae P 0b8e3ef7313048ccb808f078eea9607c: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260806 10:53:50.070943 3854 leader_election.cc:288] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d60f4b180214729a0b6402a8743b234 (127.30.218.3:46019), 0b8e3ef7313048ccb808f078eea9607c (127.30.218.1:34453)
I20260806 10:53:50.077636 3842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:50.071623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:47081 (local address 127.30.218.3:46019)
0806 10:53:50.071802 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:50.071806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:50.071823 (+ 17us) server_negotiation.cc:407] Connection header received
0806 10:53:50.072039 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:50.072044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:50.072115 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:50.072205 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:53:50.073100 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.074142 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.075913 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.076716 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.076876 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:50.076954 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:50.077242 (+ 288us) server_negotiation.cc:299] Negotiation successful
0806 10:53:50.077372 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 10:53:50.077773 3856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:50.071182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:46019 (local address 127.30.218.2:47081)
0806 10:53:50.071743 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:50.071765 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:50.071896 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:50.072374 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:50.072379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:50.072406 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:53:50.072879 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:50.072899 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.074266 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.074271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:50.075752 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:50.075765 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.076928 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.076934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:50.077091 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:50.077115 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 10:53:50.077198 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 10:53:50.077339 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":502,"threads_started":1}
I20260806 10:53:50.078084 3844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:50.072431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:48865 (local address 127.30.218.1:34453)
0806 10:53:50.072639 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:50.072645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:50.073235 (+ 590us) server_negotiation.cc:407] Connection header received
0806 10:53:50.073392 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:50.073398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:50.073495 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:50.073595 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:53:50.074099 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.074802 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.076427 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.077183 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.077327 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:50.077399 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:50.077727 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 10:53:50.077854 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 10:53:50.078431 3857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:50.072189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:34453 (local address 127.30.218.2:48865)
0806 10:53:50.073124 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:50.073143 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:50.073269 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:50.073622 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:50.073626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:50.073650 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:53:50.073964 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:50.073972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.074933 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.074939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:50.076283 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:50.076294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.077395 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.077401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:50.077586 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:50.077609 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 10:53:50.077687 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 10:53:50.078129 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":646,"threads_started":1}
I20260806 10:53:50.078784 3784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd43b1c835744d84870b92f07b9e92ae" candidate_uuid: "472a937cc01f4c688603af99d1798571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d60f4b180214729a0b6402a8743b234" is_pre_election: true
I20260806 10:53:50.079376 3784 raft_consensus.cc:2454] T bd43b1c835744d84870b92f07b9e92ae P 8d60f4b180214729a0b6402a8743b234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 472a937cc01f4c688603af99d1798571 in term 0.
I20260806 10:53:50.079389 3517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd43b1c835744d84870b92f07b9e92ae" candidate_uuid: "472a937cc01f4c688603af99d1798571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b8e3ef7313048ccb808f078eea9607c" is_pre_election: true
I20260806 10:53:50.079929 3517 raft_consensus.cc:2454] T bd43b1c835744d84870b92f07b9e92ae P 0b8e3ef7313048ccb808f078eea9607c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 472a937cc01f4c688603af99d1798571 in term 0.
I20260806 10:53:50.080246 3584 leader_election.cc:302] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 472a937cc01f4c688603af99d1798571, 8d60f4b180214729a0b6402a8743b234; no voters:
I20260806 10:53:50.080678 3854 raft_consensus.cc:2781] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:53:50.080824 3854 raft_consensus.cc:482] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:53:50.080925 3854 raft_consensus.cc:3037] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:50.083588 3854 raft_consensus.cc:504] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d60f4b180214729a0b6402a8743b234" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 46019 } } peers { permanent_uuid: "0b8e3ef7313048ccb808f078eea9607c" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34453 } } peers { permanent_uuid: "472a937cc01f4c688603af99d1798571" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 41185 } }
I20260806 10:53:50.084173 3854 leader_election.cc:288] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571 [CANDIDATE]: Term 1 election: Requested vote from peers 8d60f4b180214729a0b6402a8743b234 (127.30.218.3:46019), 0b8e3ef7313048ccb808f078eea9607c (127.30.218.1:34453)
I20260806 10:53:50.084695 3784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd43b1c835744d84870b92f07b9e92ae" candidate_uuid: "472a937cc01f4c688603af99d1798571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d60f4b180214729a0b6402a8743b234"
I20260806 10:53:50.084744 3517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd43b1c835744d84870b92f07b9e92ae" candidate_uuid: "472a937cc01f4c688603af99d1798571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b8e3ef7313048ccb808f078eea9607c"
I20260806 10:53:50.084898 3784 raft_consensus.cc:3037] T bd43b1c835744d84870b92f07b9e92ae P 8d60f4b180214729a0b6402a8743b234 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:50.084944 3517 raft_consensus.cc:3037] T bd43b1c835744d84870b92f07b9e92ae P 0b8e3ef7313048ccb808f078eea9607c [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:50.087210 3517 raft_consensus.cc:2454] T bd43b1c835744d84870b92f07b9e92ae P 0b8e3ef7313048ccb808f078eea9607c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 472a937cc01f4c688603af99d1798571 in term 1.
I20260806 10:53:50.087224 3784 raft_consensus.cc:2454] T bd43b1c835744d84870b92f07b9e92ae P 8d60f4b180214729a0b6402a8743b234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 472a937cc01f4c688603af99d1798571 in term 1.
I20260806 10:53:50.087600 3584 leader_election.cc:302] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b8e3ef7313048ccb808f078eea9607c, 472a937cc01f4c688603af99d1798571; no voters:
I20260806 10:53:50.087872 3854 raft_consensus.cc:2781] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:50.088321 3854 raft_consensus.cc:686] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571 [term 1 LEADER]: Becoming Leader. State: Replica: 472a937cc01f4c688603af99d1798571, State: Running, Role: LEADER
I20260806 10:53:50.088932 3854 consensus_queue.cc:237] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d60f4b180214729a0b6402a8743b234" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 46019 } } peers { permanent_uuid: "0b8e3ef7313048ccb808f078eea9607c" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34453 } } peers { permanent_uuid: "472a937cc01f4c688603af99d1798571" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 41185 } }
I20260806 10:53:50.093719 3373 catalog_manager.cc:5495] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571 reported cstate change: term changed from 0 to 1, leader changed from <none> to 472a937cc01f4c688603af99d1798571 (127.30.218.2). New cstate: current_term: 1 leader_uuid: "472a937cc01f4c688603af99d1798571" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d60f4b180214729a0b6402a8743b234" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 46019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b8e3ef7313048ccb808f078eea9607c" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "472a937cc01f4c688603af99d1798571" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 41185 } health_report { overall_health: HEALTHY } } }
W20260806 10:53:50.098860 3563 tablet.cc:2369] T bd43b1c835744d84870b92f07b9e92ae P 0b8e3ef7313048ccb808f078eea9607c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:53:50.114161 3846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:50.108930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54972 (local address 127.30.218.2:41185)
0806 10:53:50.109098 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:50.109102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:50.109219 (+ 117us) server_negotiation.cc:407] Connection header received
0806 10:53:50.109397 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:50.109401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:50.109512 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:50.109645 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:53:50.109959 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.110685 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.112034 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.112494 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.112621 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:50.113090 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:53:50.113681 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:53:50.113799 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:50.113864 (+ 65us) server_negotiation.cc:299] Negotiation successful
0806 10:53:50.113935 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 10:53:50.122229 3784 raft_consensus.cc:1261] T bd43b1c835744d84870b92f07b9e92ae P 8d60f4b180214729a0b6402a8743b234 [term 1 FOLLOWER]: Refusing update from remote peer 472a937cc01f4c688603af99d1798571: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:53:50.122267 3517 raft_consensus.cc:1261] T bd43b1c835744d84870b92f07b9e92ae P 0b8e3ef7313048ccb808f078eea9607c [term 1 FOLLOWER]: Refusing update from remote peer 472a937cc01f4c688603af99d1798571: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:53:50.123121 3854 consensus_queue.cc:1035] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d60f4b180214729a0b6402a8743b234" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 46019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:53:50.123436 3858 consensus_queue.cc:1035] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b8e3ef7313048ccb808f078eea9607c" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:53:50.132917 3861 mvcc.cc:205] Tried to move back new op lower bound from 7315511828970430464 to 7315511828848787456. Current Snapshot: MvccSnapshot[applied={T|T < 7315511828970430464}]
I20260806 10:53:50.150133 3372 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:53:50.150491 3372 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:53:50.154881 3372 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97603a5a930c46309c61b9a6b2d31428: Sending DeleteTablet for 3 replicas of tablet bd43b1c835744d84870b92f07b9e92ae
I20260806 10:53:50.156181 3764 tablet_service.cc:1514] Processing DeleteTablet for tablet bd43b1c835744d84870b92f07b9e92ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:53:50 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:35118
I20260806 10:53:50.156478 3496 tablet_service.cc:1514] Processing DeleteTablet for tablet bd43b1c835744d84870b92f07b9e92ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:53:50 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:55870
I20260806 10:53:50.156603 3630 tablet_service.cc:1514] Processing DeleteTablet for tablet bd43b1c835744d84870b92f07b9e92ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:53:50 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:54958
I20260806 10:53:50.158595 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 3434
I20260806 10:53:50.173029 3873 tablet_replica.cc:330] T bd43b1c835744d84870b92f07b9e92ae P 0b8e3ef7313048ccb808f078eea9607c: stopping tablet replica
I20260806 10:53:50.173660 3873 raft_consensus.cc:2229] T bd43b1c835744d84870b92f07b9e92ae P 0b8e3ef7313048ccb808f078eea9607c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:53:50.174321 3873 raft_consensus.cc:2258] T bd43b1c835744d84870b92f07b9e92ae P 0b8e3ef7313048ccb808f078eea9607c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:53:50.192082 3874 tablet_replica.cc:330] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571: stopping tablet replica
I20260806 10:53:50.192550 3874 raft_consensus.cc:2229] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:53:50.193145 3874 raft_consensus.cc:2258] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:53:50.195714 3873 ts_tablet_manager.cc:1905] T bd43b1c835744d84870b92f07b9e92ae P 0b8e3ef7313048ccb808f078eea9607c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:53:50.200232 3874 ts_tablet_manager.cc:1905] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:53:50.202033 3873 ts_tablet_manager.cc:1918] T bd43b1c835744d84870b92f07b9e92ae P 0b8e3ef7313048ccb808f078eea9607c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:53:50.202232 3873 log.cc:1196] T bd43b1c835744d84870b92f07b9e92ae P 0b8e3ef7313048ccb808f078eea9607c: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-0/wal/wals/bd43b1c835744d84870b92f07b9e92ae
I20260806 10:53:50.202751 3873 ts_tablet_manager.cc:1939] T bd43b1c835744d84870b92f07b9e92ae P 0b8e3ef7313048ccb808f078eea9607c: Deleting consensus metadata
Aug 06 10:53:50 dist-test-slave-49h3 krb5kdc[3328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.1@KRBTEST.COM
I20260806 10:53:50.202769 3872 tablet_replica.cc:330] T bd43b1c835744d84870b92f07b9e92ae P 8d60f4b180214729a0b6402a8743b234: stopping tablet replica
I20260806 10:53:50.204653 3358 catalog_manager.cc:4841] TS 0b8e3ef7313048ccb808f078eea9607c (127.30.218.1:34453): tablet bd43b1c835744d84870b92f07b9e92ae (table test-table [id=f3955b33a0134893be4970dd7d7e3aa0]) successfully deleted
I20260806 10:53:50.204993 3872 raft_consensus.cc:2229] T bd43b1c835744d84870b92f07b9e92ae P 8d60f4b180214729a0b6402a8743b234 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:53:50.205490 3872 raft_consensus.cc:2258] T bd43b1c835744d84870b92f07b9e92ae P 8d60f4b180214729a0b6402a8743b234 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:53:50.208719 3874 ts_tablet_manager.cc:1918] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:53:50.208935 3874 log.cc:1196] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/bd43b1c835744d84870b92f07b9e92ae
I20260806 10:53:50.209524 3874 ts_tablet_manager.cc:1939] T bd43b1c835744d84870b92f07b9e92ae P 472a937cc01f4c688603af99d1798571: Deleting consensus metadata
I20260806 10:53:50.211390 3358 catalog_manager.cc:4841] TS 472a937cc01f4c688603af99d1798571 (127.30.218.2:41185): tablet bd43b1c835744d84870b92f07b9e92ae (table test-table [id=f3955b33a0134893be4970dd7d7e3aa0]) successfully deleted
I20260806 10:53:50.211855 3872 ts_tablet_manager.cc:1905] T bd43b1c835744d84870b92f07b9e92ae P 8d60f4b180214729a0b6402a8743b234: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:53:50.214016 3844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:50.159998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55886 (local address 127.30.218.1:34453)
0806 10:53:50.160205 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:50.160211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:50.166186 (+ 5975us) server_negotiation.cc:407] Connection header received
0806 10:53:50.167191 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:50.167199 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:50.167315 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:50.167413 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:50.174942 (+ 7529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.176248 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.193817 (+ 17569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.194390 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.194477 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:50.206508 (+ 12031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:50.206523 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:50.206540 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:50.206583 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:50.210462 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:50.211471 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:50.211481 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:50.211485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:50.211723 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:50.212221 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:50.212233 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:50.212236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:50.212708 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:50.213005 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:50.213372 (+ 367us) server_negotiation.cc:299] Negotiation successful
0806 10:53:50.213645 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 10:53:50.214727 3557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:50.224272 3872 ts_tablet_manager.cc:1918] T bd43b1c835744d84870b92f07b9e92ae P 8d60f4b180214729a0b6402a8743b234: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:53:50.224507 3872 log.cc:1196] T bd43b1c835744d84870b92f07b9e92ae P 8d60f4b180214729a0b6402a8743b234: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/bd43b1c835744d84870b92f07b9e92ae
I20260806 10:53:50.225165 3872 ts_tablet_manager.cc:1939] T bd43b1c835744d84870b92f07b9e92ae P 8d60f4b180214729a0b6402a8743b234: Deleting consensus metadata
I20260806 10:53:50.227394 3358 catalog_manager.cc:4841] TS 8d60f4b180214729a0b6402a8743b234 (127.30.218.3:46019): tablet bd43b1c835744d84870b92f07b9e92ae (table test-table [id=f3955b33a0134893be4970dd7d7e3aa0]) successfully deleted
I20260806 10:53:50.306133 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 3434
I20260806 10:53:50.322809 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 3568
Aug 06 10:53:50 dist-test-slave-49h3 krb5kdc[3328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.2@KRBTEST.COM
I20260806 10:53:50.341550 3846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:50.323706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54988 (local address 127.30.218.2:41185)
0806 10:53:50.323933 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:50.323940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:50.323962 (+ 22us) server_negotiation.cc:407] Connection header received
0806 10:53:50.324063 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:50.324070 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:50.324160 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:50.324279 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:50.326000 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.327265 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.328587 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.329278 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.329421 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:50.333830 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:50.333854 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:50.333875 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:50.333933 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:50.338195 (+ 4262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:50.339022 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:50.339032 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:50.339036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:50.339262 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:50.339669 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:50.339674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:50.339678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:50.340197 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:50.340530 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:50.340835 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 10:53:50.341092 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 10:53:50.342371 3691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:50.457743 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 3568
I20260806 10:53:50.476003 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 3702
Aug 06 10:53:50 dist-test-slave-49h3 krb5kdc[3328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.3@KRBTEST.COM
I20260806 10:53:50.491562 3842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:50.476801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35134 (local address 127.30.218.3:46019)
0806 10:53:50.477007 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:50.477012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:50.477041 (+ 29us) server_negotiation.cc:407] Connection header received
0806 10:53:50.477132 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:50.477136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:50.477211 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:50.477295 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:50.478691 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.479496 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.480914 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:50.481339 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:50.481609 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:50.486057 (+ 4448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:50.486078 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:50.486097 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:50.486141 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:50.488888 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:50.489651 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:50.489660 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:50.489663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:50.489790 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:50.490137 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:50.490140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:50.490142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:50.490530 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:50.490751 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:50.491020 (+ 269us) server_negotiation.cc:299] Negotiation successful
0806 10:53:50.491235 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 10:53:50.492220 3824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:50.583583 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 3702
I20260806 10:53:50.602484 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 3342
I20260806 10:53:50.603735 3403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:50.715510 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 3342
2026-08-06T10:53:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4285 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T10:53:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:53:50Z Disabled control of system clock
I20260806 10:53:50.793974 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:46631
--webserver_interface=127.30.218.62
--webserver_port=0
--builtin_ntp_servers=127.30.218.20:40061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:46631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 10:53:51.140332 3883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:51.140673 3883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:51.140772 3883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:51.148675 3883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:53:51.148815 3883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:51.148854 3883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:51.148913 3883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:53:51.148959 3883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:53:51.159421 3883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:40061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:46631
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:46631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:51.161041 3883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:51.163091 3883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:51.175881 3888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:51.175922 3889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:51.175948 3891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:51.176970 3883 server_base.cc:1034] running on GCE node
I20260806 10:53:51.177944 3883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:51.179556 3883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:51.180835 3883 hybrid_clock.cc:648] HybridClock initialized: now 1786013631180732 us; error 87 us; skew 500 ppm
I20260806 10:53:51.181252 3883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:53:51.184487 3883 webserver.cc:460] Webserver started at http://127.30.218.62:43571/ using document root <none> and password file <none>
I20260806 10:53:51.185312 3883 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:51.185432 3883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:51.185833 3883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:51.189201 3883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "abd22a5c3e5f4a1da4b915194072f733"
format_stamp: "Formatted at 2026-08-06 10:53:51 on dist-test-slave-49h3"
server_key: "c741087cd26264bb935d5904e9e38779"
server_key_iv: "b48785d7b91acafde61c9ee1619c7ce5"
server_key_version: "encryptionkey@0"
I20260806 10:53:51.190209 3883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "abd22a5c3e5f4a1da4b915194072f733"
format_stamp: "Formatted at 2026-08-06 10:53:51 on dist-test-slave-49h3"
server_key: "c741087cd26264bb935d5904e9e38779"
server_key_iv: "b48785d7b91acafde61c9ee1619c7ce5"
server_key_version: "encryptionkey@0"
I20260806 10:53:51.196333 3883 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260806 10:53:51.200583 3897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:51.202181 3883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:53:51.202394 3883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "abd22a5c3e5f4a1da4b915194072f733"
format_stamp: "Formatted at 2026-08-06 10:53:51 on dist-test-slave-49h3"
server_key: "c741087cd26264bb935d5904e9e38779"
server_key_iv: "b48785d7b91acafde61c9ee1619c7ce5"
server_key_version: "encryptionkey@0"
I20260806 10:53:51.202656 3883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:51.209057 3883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:51.209969 3883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:51.210289 3883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:51.228036 3883 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:46631
I20260806 10:53:51.228058 3949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:46631 every 8 connection(s)
I20260806 10:53:51.229727 3883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:53:51.233824 3950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:51.239437 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 3883
I20260806 10:53:51.239707 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:53:51.240473 31592 external_mini_cluster.cc:1428] Setting key ed6b2256f8484e91b977732ec3c9ad53
I20260806 10:53:51.246279 3950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abd22a5c3e5f4a1da4b915194072f733: Bootstrap starting.
I20260806 10:53:51.249302 3953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:51.242795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34632 (local address 127.30.218.62:46631)
0806 10:53:51.243890 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:51.243905 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:51.243952 (+ 47us) server_negotiation.cc:407] Connection header received
0806 10:53:51.245723 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:51.245746 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:51.246187 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:51.246559 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:51.247667 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:51.247677 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:51.247688 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:53:51.247707 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:51.247772 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:53:51.247794 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:51.248006 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:51.248139 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 10:53:51.248407 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":383,"threads_started":1}
I20260806 10:53:51.249635 3950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abd22a5c3e5f4a1da4b915194072f733: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:51.250746 3950 log.cc:824] T 00000000000000000000000000000000 P abd22a5c3e5f4a1da4b915194072f733: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:51.253589 3950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abd22a5c3e5f4a1da4b915194072f733: No bootstrap required, opened a new log
I20260806 10:53:51.261011 3950 raft_consensus.cc:348] T 00000000000000000000000000000000 P abd22a5c3e5f4a1da4b915194072f733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abd22a5c3e5f4a1da4b915194072f733" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46631 } }
I20260806 10:53:51.261435 3950 raft_consensus.cc:374] T 00000000000000000000000000000000 P abd22a5c3e5f4a1da4b915194072f733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:51.261574 3950 raft_consensus.cc:729] T 00000000000000000000000000000000 P abd22a5c3e5f4a1da4b915194072f733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abd22a5c3e5f4a1da4b915194072f733, State: Initialized, Role: FOLLOWER
I20260806 10:53:51.262162 3950 consensus_queue.cc:260] T 00000000000000000000000000000000 P abd22a5c3e5f4a1da4b915194072f733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abd22a5c3e5f4a1da4b915194072f733" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46631 } }
I20260806 10:53:51.262389 3950 raft_consensus.cc:388] T 00000000000000000000000000000000 P abd22a5c3e5f4a1da4b915194072f733 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:53:51.262517 3950 raft_consensus.cc:482] T 00000000000000000000000000000000 P abd22a5c3e5f4a1da4b915194072f733 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:53:51.262732 3950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abd22a5c3e5f4a1da4b915194072f733 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:51.265120 3950 raft_consensus.cc:504] T 00000000000000000000000000000000 P abd22a5c3e5f4a1da4b915194072f733 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abd22a5c3e5f4a1da4b915194072f733" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46631 } }
I20260806 10:53:51.265700 3950 leader_election.cc:302] T 00000000000000000000000000000000 P abd22a5c3e5f4a1da4b915194072f733 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abd22a5c3e5f4a1da4b915194072f733; no voters:
I20260806 10:53:51.266388 3950 leader_election.cc:288] T 00000000000000000000000000000000 P abd22a5c3e5f4a1da4b915194072f733 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:53:51.266533 3955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abd22a5c3e5f4a1da4b915194072f733 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:51.267488 3955 raft_consensus.cc:686] T 00000000000000000000000000000000 P abd22a5c3e5f4a1da4b915194072f733 [term 1 LEADER]: Becoming Leader. State: Replica: abd22a5c3e5f4a1da4b915194072f733, State: Running, Role: LEADER
I20260806 10:53:51.268064 3955 consensus_queue.cc:237] T 00000000000000000000000000000000 P abd22a5c3e5f4a1da4b915194072f733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abd22a5c3e5f4a1da4b915194072f733" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46631 } }
I20260806 10:53:51.268754 3950 sys_catalog.cc:564] T 00000000000000000000000000000000 P abd22a5c3e5f4a1da4b915194072f733 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:53:51.271138 3957 sys_catalog.cc:455] T 00000000000000000000000000000000 P abd22a5c3e5f4a1da4b915194072f733 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abd22a5c3e5f4a1da4b915194072f733. Latest consensus state: current_term: 1 leader_uuid: "abd22a5c3e5f4a1da4b915194072f733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abd22a5c3e5f4a1da4b915194072f733" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46631 } } }
I20260806 10:53:51.271135 3956 sys_catalog.cc:455] T 00000000000000000000000000000000 P abd22a5c3e5f4a1da4b915194072f733 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abd22a5c3e5f4a1da4b915194072f733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abd22a5c3e5f4a1da4b915194072f733" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46631 } } }
I20260806 10:53:51.271646 3957 sys_catalog.cc:458] T 00000000000000000000000000000000 P abd22a5c3e5f4a1da4b915194072f733 [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:51.271654 3956 sys_catalog.cc:458] T 00000000000000000000000000000000 P abd22a5c3e5f4a1da4b915194072f733 [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:51.274248 3963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:53:51.278812 3963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:53:51.289288 3963 catalog_manager.cc:1347] Generated new cluster ID: fe5b71aa67f44d8db56cc76903a76cfc
I20260806 10:53:51.289430 3963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:53:51.315774 3963 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:53:51.316368 3963 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:53:51.327334 3963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abd22a5c3e5f4a1da4b915194072f733: Generated new TSK 0
I20260806 10:53:51.328462 3963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:53:51.335727 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:46631
--builtin_ntp_servers=127.30.218.20:40061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 10:53:51.679154 3974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:51.679481 3974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:51.679628 3974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:51.687353 3974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:51.687475 3974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:51.687619 3974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:53:51.696900 3974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:40061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:46631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:51.698302 3974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:51.700274 3974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:51.713631 3980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:51.713596 3979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:51.713662 3982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:51.714500 3974 server_base.cc:1034] running on GCE node
I20260806 10:53:51.715127 3974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:51.716210 3974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:51.717429 3974 hybrid_clock.cc:648] HybridClock initialized: now 1786013631717369 us; error 54 us; skew 500 ppm
I20260806 10:53:51.717818 3974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:53:51.720255 3974 webserver.cc:460] Webserver started at http://127.30.218.1:37709/ using document root <none> and password file <none>
I20260806 10:53:51.720997 3974 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:51.721107 3974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:51.721439 3974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:51.724591 3974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "dc70b50a5c82417b8c45a9c453021db9"
format_stamp: "Formatted at 2026-08-06 10:53:51 on dist-test-slave-49h3"
server_key: "b95acb46b7d9a170ec2596eff7226e12"
server_key_iv: "2035803772e02d3eff6e1b0a691a7af3"
server_key_version: "encryptionkey@0"
I20260806 10:53:51.725483 3974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "dc70b50a5c82417b8c45a9c453021db9"
format_stamp: "Formatted at 2026-08-06 10:53:51 on dist-test-slave-49h3"
server_key: "b95acb46b7d9a170ec2596eff7226e12"
server_key_iv: "2035803772e02d3eff6e1b0a691a7af3"
server_key_version: "encryptionkey@0"
I20260806 10:53:51.731164 3974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 10:53:51.735172 3988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:51.736677 3974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:53:51.736855 3974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "dc70b50a5c82417b8c45a9c453021db9"
format_stamp: "Formatted at 2026-08-06 10:53:51 on dist-test-slave-49h3"
server_key: "b95acb46b7d9a170ec2596eff7226e12"
server_key_iv: "2035803772e02d3eff6e1b0a691a7af3"
server_key_version: "encryptionkey@0"
I20260806 10:53:51.737088 3974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:51.743001 3974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:51.743911 3974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:51.744201 3974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:51.745209 3974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:53:51.746874 3974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:51.746980 3974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:51.747102 3974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:51.747177 3974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:51.779642 3974 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:38777
I20260806 10:53:51.779711 4101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:38777 every 8 connection(s)
I20260806 10:53:51.781054 3974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:53:51.789275 4104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:51.783675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:38323 (local address 127.30.218.62:46631)
0806 10:53:51.784188 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:51.784193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:51.785374 (+ 1181us) server_negotiation.cc:407] Connection header received
0806 10:53:51.787116 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:51.787121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:51.787190 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:51.787287 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:51.788527 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:51.788531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:51.788534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:53:51.788551 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:51.788600 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:53:51.788613 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:51.788720 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:51.788912 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 10:53:51.788983 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":279,"threads_started":1}
I20260806 10:53:51.790184 4105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:51.784282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46631 (local address 127.30.218.1:38323)
0806 10:53:51.785223 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:51.785263 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:51.786852 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:51.787526 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:51.787535 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:51.787627 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:51.788228 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:53:51.788230 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:51.788268 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:53:51.788352 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:51.788702 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:51.788714 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:51.788721 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 10:53:51.788974 (+ 253us) client_negotiation.cc:240] Negotiation successful
0806 10:53:51.789227 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":265,"threads_started":1}
I20260806 10:53:51.791273 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 3974
I20260806 10:53:51.791491 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:53:51.791859 4102 heartbeater.cc:347] Connected to a master server at 127.30.218.62:46631
I20260806 10:53:51.792164 4102 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:51.792232 31592 external_mini_cluster.cc:1428] Setting key 9370e16c9df38b5ac60fbcc5dd084438
I20260806 10:53:51.792783 4102 heartbeater.cc:511] Master 127.30.218.62:46631 requested a full tablet report, sending...
I20260806 10:53:51.794968 3915 ts_manager.cc:194] Registered new tserver with Master: dc70b50a5c82417b8c45a9c453021db9 (127.30.218.1:38777)
I20260806 10:53:51.794997 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:46631
--builtin_ntp_servers=127.30.218.20:40061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 10:53:52.137883 4106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:52.138303 4106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:52.138422 4106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:52.146862 4106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:52.147011 4106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:52.147140 4106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:53:52.157495 4106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:40061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:46631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:52.159345 4106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:52.161628 4106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:52.177002 4111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:52.177038 4112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:52.176955 4114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:52.178438 4106 server_base.cc:1034] running on GCE node
I20260806 10:53:52.179143 4106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:52.180395 4106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:52.181624 4106 hybrid_clock.cc:648] HybridClock initialized: now 1786013632181558 us; error 58 us; skew 500 ppm
I20260806 10:53:52.181986 4106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:53:52.184753 4106 webserver.cc:460] Webserver started at http://127.30.218.2:38533/ using document root <none> and password file <none>
I20260806 10:53:52.185657 4106 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:52.185761 4106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:52.186169 4106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:52.189628 4106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "dac00de4c5fe4d7e8ab9209fe9930175"
format_stamp: "Formatted at 2026-08-06 10:53:52 on dist-test-slave-49h3"
server_key: "d12845f63cd491e56dc01b34c0791198"
server_key_iv: "619ad027557865c7c01cc013481707a7"
server_key_version: "encryptionkey@0"
I20260806 10:53:52.190495 4106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "dac00de4c5fe4d7e8ab9209fe9930175"
format_stamp: "Formatted at 2026-08-06 10:53:52 on dist-test-slave-49h3"
server_key: "d12845f63cd491e56dc01b34c0791198"
server_key_iv: "619ad027557865c7c01cc013481707a7"
server_key_version: "encryptionkey@0"
I20260806 10:53:52.196740 4106 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 10:53:52.201771 4120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:52.203681 4106 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 10:53:52.203907 4106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "dac00de4c5fe4d7e8ab9209fe9930175"
format_stamp: "Formatted at 2026-08-06 10:53:52 on dist-test-slave-49h3"
server_key: "d12845f63cd491e56dc01b34c0791198"
server_key_iv: "619ad027557865c7c01cc013481707a7"
server_key_version: "encryptionkey@0"
I20260806 10:53:52.204183 4106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:52.211176 4106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:52.212146 4106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:52.212445 4106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:52.213527 4106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:53:52.215279 4106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:52.215395 4106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:52.215507 4106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:52.215579 4106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:52.250062 4106 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:35191
I20260806 10:53:52.250102 4233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:35191 every 8 connection(s)
I20260806 10:53:52.251962 4106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:53:52.261361 4104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:52.254974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:34125 (local address 127.30.218.62:46631)
0806 10:53:52.255215 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:52.255220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:52.256878 (+ 1658us) server_negotiation.cc:407] Connection header received
0806 10:53:52.258869 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:52.258876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:52.258974 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:52.259072 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:52.260538 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:52.260543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:52.260547 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:53:52.260566 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:52.260606 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:53:52.260619 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:52.260747 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:52.260929 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 10:53:52.261016 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 10:53:52.261659 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 4106
I20260806 10:53:52.261883 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:53:52.262395 31592 external_mini_cluster.cc:1428] Setting key fb026fdc16febbcf47ea311eea533bb2
I20260806 10:53:52.262493 4236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:52.255595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46631 (local address 127.30.218.2:34125)
0806 10:53:52.256698 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:52.256740 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:52.258569 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:52.259340 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:52.259349 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:52.259451 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:52.260193 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:53:52.260196 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:52.260239 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:53:52.260354 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:52.260711 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:52.260723 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:52.260732 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 10:53:52.261010 (+ 278us) client_negotiation.cc:240] Negotiation successful
0806 10:53:52.261298 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":310,"threads_started":1}
I20260806 10:53:52.264231 4234 heartbeater.cc:347] Connected to a master server at 127.30.218.62:46631
I20260806 10:53:52.264597 4234 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:52.265364 4234 heartbeater.cc:511] Master 127.30.218.62:46631 requested a full tablet report, sending...
I20260806 10:53:52.265900 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:46631
--builtin_ntp_servers=127.30.218.20:40061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 10:53:52.267028 3915 ts_manager.cc:194] Registered new tserver with Master: dac00de4c5fe4d7e8ab9209fe9930175 (127.30.218.2:35191)
W20260806 10:53:52.662492 4237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:52.662889 4237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:52.662992 4237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:52.671131 4237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:52.671288 4237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:52.671429 4237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:53:52.681968 4237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:40061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:46631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:52.683706 4237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:52.686061 4237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:52.701939 4242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:52.702075 4243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:52.702502 4245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:52.703390 4237 server_base.cc:1034] running on GCE node
I20260806 10:53:52.704241 4237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:52.705775 4237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:52.707094 4237 hybrid_clock.cc:648] HybridClock initialized: now 1786013632706991 us; error 85 us; skew 500 ppm
I20260806 10:53:52.707559 4237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:53:52.710824 4237 webserver.cc:460] Webserver started at http://127.30.218.3:44887/ using document root <none> and password file <none>
I20260806 10:53:52.711769 4237 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:52.711887 4237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:52.712247 4237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:52.715916 4237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "26f5b14b786047d1a6b008e650d7872c"
format_stamp: "Formatted at 2026-08-06 10:53:52 on dist-test-slave-49h3"
server_key: "cfafd985ccd4641aa3f69e52968982ef"
server_key_iv: "b9cf2037334944a0a3938630a9eda8a4"
server_key_version: "encryptionkey@0"
I20260806 10:53:52.716872 4237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "26f5b14b786047d1a6b008e650d7872c"
format_stamp: "Formatted at 2026-08-06 10:53:52 on dist-test-slave-49h3"
server_key: "cfafd985ccd4641aa3f69e52968982ef"
server_key_iv: "b9cf2037334944a0a3938630a9eda8a4"
server_key_version: "encryptionkey@0"
I20260806 10:53:52.723196 4237 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 10:53:52.727911 4251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:52.729800 4237 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 10:53:52.730032 4237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "26f5b14b786047d1a6b008e650d7872c"
format_stamp: "Formatted at 2026-08-06 10:53:52 on dist-test-slave-49h3"
server_key: "cfafd985ccd4641aa3f69e52968982ef"
server_key_iv: "b9cf2037334944a0a3938630a9eda8a4"
server_key_version: "encryptionkey@0"
I20260806 10:53:52.730312 4237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:52.737299 4237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:52.738581 4237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:52.738898 4237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:52.740167 4237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:53:52.742250 4237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:53:52.742372 4237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:52.742538 4237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:53:52.742626 4237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:52.792055 4237 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:43615
I20260806 10:53:52.792138 4364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:43615 every 8 connection(s)
I20260806 10:53:52.793967 4237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:53:52.798092 4102 heartbeater.cc:503] Master 127.30.218.62:46631 was elected leader, sending a full tablet report...
I20260806 10:53:52.799088 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 4237
I20260806 10:53:52.799341 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:53:52.800041 31592 external_mini_cluster.cc:1428] Setting key e585f3afe6fe4e3089dcb478bca3a8c5
I20260806 10:53:52.805828 4367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:52.797307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:43823 (local address 127.30.218.62:46631)
0806 10:53:52.797966 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:52.797975 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:52.799647 (+ 1672us) server_negotiation.cc:407] Connection header received
0806 10:53:52.802535 (+ 2888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:52.802543 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:52.802632 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:52.802737 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:52.804941 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:52.804949 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:52.804954 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:53:52.804979 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:52.805043 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:53:52.805060 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:52.805184 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:52.805416 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 10:53:52.805523 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":291,"threads_started":1}
I20260806 10:53:52.807475 4368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:52.798098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46631 (local address 127.30.218.3:43823)
0806 10:53:52.799449 (+ 1351us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:52.799515 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:52.802149 (+ 2634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:52.803409 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:52.803423 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:52.803573 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:52.804507 (+ 934us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:53:52.804511 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:52.804577 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:53:52.804725 (+ 148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:52.805170 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:52.805181 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:52.805190 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 10:53:52.805571 (+ 381us) client_negotiation.cc:240] Negotiation successful
0806 10:53:52.805933 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":389,"threads_started":1}
I20260806 10:53:52.809278 4365 heartbeater.cc:347] Connected to a master server at 127.30.218.62:46631
I20260806 10:53:52.809725 4365 heartbeater.cc:464] Registering TS with master...
I20260806 10:53:52.810515 4365 heartbeater.cc:511] Master 127.30.218.62:46631 requested a full tablet report, sending...
I20260806 10:53:52.812278 3915 ts_manager.cc:194] Registered new tserver with Master: 26f5b14b786047d1a6b008e650d7872c (127.30.218.3:43615)
I20260806 10:53:52.816068 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:53:52.824023 4367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:52.821157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34648 (local address 127.30.218.62:46631)
0806 10:53:52.821343 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:52.821348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:52.821912 (+ 564us) server_negotiation.cc:407] Connection header received
0806 10:53:52.822119 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:52.822125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:52.822196 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:52.822285 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:52.823318 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:52.823325 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:52.823330 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:53:52.823348 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:52.823390 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:53:52.823403 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:52.823541 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:52.823636 (+ 95us) server_negotiation.cc:299] Negotiation successful
0806 10:53:52.823712 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 10:53:52.829032 3915 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:53:52.837426 3915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:53:52.858883 4380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:52.852607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57072 (local address 127.30.218.1:38777)
0806 10:53:52.853490 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:52.853501 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:52.854113 (+ 612us) server_negotiation.cc:407] Connection header received
0806 10:53:52.854451 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:52.854461 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:52.854757 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:52.854935 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:52.857794 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:52.857804 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:52.857819 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:53:52.857839 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:52.857911 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:53:52.857938 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:52.858141 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:52.858322 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 10:53:52.858439 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":474,"threads_started":1}
I20260806 10:53:52.859247 4378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:52.852583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:35191 (local address 127.0.0.1:59460)
0806 10:53:52.854396 (+ 1813us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:52.854413 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:52.854632 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:52.856262 (+ 1630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:52.856271 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:52.856390 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:52.857487 (+ 1097us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:53:52.857490 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:52.857525 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:53:52.857620 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:52.858251 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:52.858255 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:52.858264 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 10:53:52.858406 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 10:53:52.858472 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1658,"thread_start_us":921,"threads_started":1}
I20260806 10:53:52.859740 4381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:52.853146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59460 (local address 127.30.218.2:35191)
0806 10:53:52.853912 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:52.853922 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:52.854529 (+ 607us) server_negotiation.cc:407] Connection header received
0806 10:53:52.854800 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:52.854808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:52.855033 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:52.855164 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:52.857924 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:52.857930 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:52.857940 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:53:52.857956 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:52.858042 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:53:52.858066 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:52.858206 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:52.859296 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0806 10:53:52.859412 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":486,"threads_started":1}
I20260806 10:53:52.860069 4379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:52.852362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:38777 (local address 127.0.0.1:57072)
0806 10:53:52.853949 (+ 1587us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:52.853999 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:52.854264 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:52.856278 (+ 2014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:52.856283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:52.856390 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:52.857510 (+ 1120us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:53:52.857514 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:52.857530 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:53:52.857620 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:52.858058 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:52.858070 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:52.858091 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 10:53:52.858297 (+ 206us) client_negotiation.cc:240] Negotiation successful
0806 10:53:52.858411 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1308,"thread_start_us":1734,"threads_started":1}
I20260806 10:53:52.860880 4382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:52.854432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:43615 (local address 127.0.0.1:40136)
0806 10:53:52.857383 (+ 2951us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:52.857723 (+ 340us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:52.857787 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:52.858588 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:52.858592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:52.858613 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:52.859626 (+ 1013us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:53:52.859629 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:52.859644 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:53:52.859684 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:52.860251 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:52.860257 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:52.860261 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 10:53:52.860406 (+ 145us) client_negotiation.cc:240] Negotiation successful
0806 10:53:52.860458 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2790,"thread_start_us":2656,"threads_started":1}
I20260806 10:53:52.863883 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:52.854852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40136 (local address 127.30.218.3:43615)
0806 10:53:52.857891 (+ 3039us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:52.857898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:52.857939 (+ 41us) server_negotiation.cc:407] Connection header received
0806 10:53:52.858027 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:52.858035 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:52.858228 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:52.858389 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:52.859889 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:52.859898 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:52.859910 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:53:52.859929 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:52.860006 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:53:52.860028 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:52.860191 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:52.861152 (+ 961us) server_negotiation.cc:299] Negotiation successful
0806 10:53:52.863464 (+ 2312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2423,"thread_start_us":2559,"threads_started":1}
I20260806 10:53:52.863916 4037 tablet_service.cc:1467] Processing CreateTablet for tablet 38d1caa2f7024126b654c4e5ad841e96 (DEFAULT_TABLE table=test-table [id=0cef38fc74224d0fb4abcfdf5a337a89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:52.863914 4169 tablet_service.cc:1467] Processing CreateTablet for tablet 38d1caa2f7024126b654c4e5ad841e96 (DEFAULT_TABLE table=test-table [id=0cef38fc74224d0fb4abcfdf5a337a89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:52.865468 4037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d1caa2f7024126b654c4e5ad841e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:52.865500 4169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d1caa2f7024126b654c4e5ad841e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:52.867636 4300 tablet_service.cc:1467] Processing CreateTablet for tablet 38d1caa2f7024126b654c4e5ad841e96 (DEFAULT_TABLE table=test-table [id=0cef38fc74224d0fb4abcfdf5a337a89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:53:52.869166 4300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d1caa2f7024126b654c4e5ad841e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:52.878343 4384 tablet_bootstrap.cc:492] T 38d1caa2f7024126b654c4e5ad841e96 P dc70b50a5c82417b8c45a9c453021db9: Bootstrap starting.
I20260806 10:53:52.880843 4385 tablet_bootstrap.cc:492] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175: Bootstrap starting.
I20260806 10:53:52.884240 4384 tablet_bootstrap.cc:654] T 38d1caa2f7024126b654c4e5ad841e96 P dc70b50a5c82417b8c45a9c453021db9: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:52.884917 4386 tablet_bootstrap.cc:492] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c: Bootstrap starting.
I20260806 10:53:52.886359 4385 tablet_bootstrap.cc:654] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:52.886685 4384 log.cc:824] T 38d1caa2f7024126b654c4e5ad841e96 P dc70b50a5c82417b8c45a9c453021db9: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:52.887849 4385 log.cc:824] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:52.889874 4386 tablet_bootstrap.cc:654] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:52.891225 4386 log.cc:824] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:52.891402 4385 tablet_bootstrap.cc:492] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175: No bootstrap required, opened a new log
I20260806 10:53:52.891707 4385 ts_tablet_manager.cc:1397] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260806 10:53:52.891767 4384 tablet_bootstrap.cc:492] T 38d1caa2f7024126b654c4e5ad841e96 P dc70b50a5c82417b8c45a9c453021db9: No bootstrap required, opened a new log
I20260806 10:53:52.892144 4384 ts_tablet_manager.cc:1397] T 38d1caa2f7024126b654c4e5ad841e96 P dc70b50a5c82417b8c45a9c453021db9: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.005s
I20260806 10:53:52.894881 4386 tablet_bootstrap.cc:492] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c: No bootstrap required, opened a new log
I20260806 10:53:52.895253 4386 ts_tablet_manager.cc:1397] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20260806 10:53:52.900499 4385 raft_consensus.cc:348] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc70b50a5c82417b8c45a9c453021db9" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38777 } } peers { permanent_uuid: "dac00de4c5fe4d7e8ab9209fe9930175" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35191 } } peers { permanent_uuid: "26f5b14b786047d1a6b008e650d7872c" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 43615 } }
I20260806 10:53:52.900960 4385 raft_consensus.cc:374] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:52.901067 4385 raft_consensus.cc:729] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dac00de4c5fe4d7e8ab9209fe9930175, State: Initialized, Role: FOLLOWER
I20260806 10:53:52.901837 4385 consensus_queue.cc:260] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc70b50a5c82417b8c45a9c453021db9" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38777 } } peers { permanent_uuid: "dac00de4c5fe4d7e8ab9209fe9930175" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35191 } } peers { permanent_uuid: "26f5b14b786047d1a6b008e650d7872c" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 43615 } }
I20260806 10:53:52.903191 4234 heartbeater.cc:503] Master 127.30.218.62:46631 was elected leader, sending a full tablet report...
I20260806 10:53:52.903025 4384 raft_consensus.cc:348] T 38d1caa2f7024126b654c4e5ad841e96 P dc70b50a5c82417b8c45a9c453021db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc70b50a5c82417b8c45a9c453021db9" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38777 } } peers { permanent_uuid: "dac00de4c5fe4d7e8ab9209fe9930175" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35191 } } peers { permanent_uuid: "26f5b14b786047d1a6b008e650d7872c" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 43615 } }
I20260806 10:53:52.903606 4384 raft_consensus.cc:374] T 38d1caa2f7024126b654c4e5ad841e96 P dc70b50a5c82417b8c45a9c453021db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:52.903730 4384 raft_consensus.cc:729] T 38d1caa2f7024126b654c4e5ad841e96 P dc70b50a5c82417b8c45a9c453021db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc70b50a5c82417b8c45a9c453021db9, State: Initialized, Role: FOLLOWER
I20260806 10:53:52.903975 4385 ts_tablet_manager.cc:1428] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260806 10:53:52.904636 4384 consensus_queue.cc:260] T 38d1caa2f7024126b654c4e5ad841e96 P dc70b50a5c82417b8c45a9c453021db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc70b50a5c82417b8c45a9c453021db9" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38777 } } peers { permanent_uuid: "dac00de4c5fe4d7e8ab9209fe9930175" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35191 } } peers { permanent_uuid: "26f5b14b786047d1a6b008e650d7872c" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 43615 } }
I20260806 10:53:52.907125 4384 ts_tablet_manager.cc:1428] T 38d1caa2f7024126b654c4e5ad841e96 P dc70b50a5c82417b8c45a9c453021db9: Time spent starting tablet: real 0.015s user 0.014s sys 0.002s
I20260806 10:53:52.908293 4386 raft_consensus.cc:348] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc70b50a5c82417b8c45a9c453021db9" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38777 } } peers { permanent_uuid: "dac00de4c5fe4d7e8ab9209fe9930175" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35191 } } peers { permanent_uuid: "26f5b14b786047d1a6b008e650d7872c" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 43615 } }
I20260806 10:53:52.908944 4386 raft_consensus.cc:374] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:52.909085 4386 raft_consensus.cc:729] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f5b14b786047d1a6b008e650d7872c, State: Initialized, Role: FOLLOWER
I20260806 10:53:52.910060 4386 consensus_queue.cc:260] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc70b50a5c82417b8c45a9c453021db9" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38777 } } peers { permanent_uuid: "dac00de4c5fe4d7e8ab9209fe9930175" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35191 } } peers { permanent_uuid: "26f5b14b786047d1a6b008e650d7872c" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 43615 } }
I20260806 10:53:52.911536 4365 heartbeater.cc:503] Master 127.30.218.62:46631 was elected leader, sending a full tablet report...
I20260806 10:53:52.912411 4386 ts_tablet_manager.cc:1428] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c: Time spent starting tablet: real 0.017s user 0.012s sys 0.002s
W20260806 10:53:53.004410 4235 tablet.cc:2369] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:53:53.033913 4103 tablet.cc:2369] T 38d1caa2f7024126b654c4e5ad841e96 P dc70b50a5c82417b8c45a9c453021db9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:53:53.046227 4366 tablet.cc:2369] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:53:53.239490 4393 raft_consensus.cc:482] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:53:53.239933 4393 raft_consensus.cc:504] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc70b50a5c82417b8c45a9c453021db9" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38777 } } peers { permanent_uuid: "dac00de4c5fe4d7e8ab9209fe9930175" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35191 } } peers { permanent_uuid: "26f5b14b786047d1a6b008e650d7872c" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 43615 } }
I20260806 10:53:53.242267 4390 raft_consensus.cc:482] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:53:53.242547 4393 leader_election.cc:288] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc70b50a5c82417b8c45a9c453021db9 (127.30.218.1:38777), dac00de4c5fe4d7e8ab9209fe9930175 (127.30.218.2:35191)
I20260806 10:53:53.242712 4390 raft_consensus.cc:504] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc70b50a5c82417b8c45a9c453021db9" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38777 } } peers { permanent_uuid: "dac00de4c5fe4d7e8ab9209fe9930175" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35191 } } peers { permanent_uuid: "26f5b14b786047d1a6b008e650d7872c" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 43615 } }
I20260806 10:53:53.246083 4368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:53.242889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:38777 (local address 127.30.218.3:47017)
0806 10:53:53.243183 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:53.243204 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:53.243341 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:53.244208 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:53.244214 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:53.244244 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:53.244969 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:53:53.244972 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:53.244996 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:53:53.245057 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:53.245582 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:53.245588 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:53.245593 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 10:53:53.245709 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 10:53:53.245772 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260806 10:53:53.246289 4381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:53.243795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:44509 (local address 127.30.218.2:35191)
0806 10:53:53.244057 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:53.244062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:53.244077 (+ 15us) server_negotiation.cc:407] Connection header received
0806 10:53:53.244136 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:53.244141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:53.244200 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:53.244293 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:53.245334 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:53.245340 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:53.245345 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:53:53.245381 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:53.245422 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:53:53.245440 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:53.245578 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:53.245830 (+ 252us) server_negotiation.cc:299] Negotiation successful
0806 10:53:53.245911 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260806 10:53:53.246758 4380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:53.242902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:47017 (local address 127.30.218.1:38777)
0806 10:53:53.243117 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:53.243123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:53.243273 (+ 150us) server_negotiation.cc:407] Connection header received
0806 10:53:53.243548 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:53.243556 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:53.243639 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:53.243738 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:53.245261 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:53.245269 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:53.245274 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:53:53.245298 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:53.245346 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:53:53.245365 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:53.245521 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:53.246381 (+ 860us) server_negotiation.cc:299] Negotiation successful
0806 10:53:53.246459 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 10:53:53.248027 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:53.243194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:35191 (local address 127.30.218.3:44509)
0806 10:53:53.243773 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:53.243791 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:53.243890 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:53.244326 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:53.244331 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:53.244354 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:53.245078 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:53:53.245081 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:53.245095 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:53:53.245143 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:53.245684 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:53.245689 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:53.245694 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 10:53:53.246960 (+ 1266us) client_negotiation.cc:240] Negotiation successful
0806 10:53:53.247028 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":672,"threads_started":1}
I20260806 10:53:53.248303 4057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d1caa2f7024126b654c4e5ad841e96" candidate_uuid: "26f5b14b786047d1a6b008e650d7872c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc70b50a5c82417b8c45a9c453021db9" is_pre_election: true
I20260806 10:53:53.249110 4057 raft_consensus.cc:2454] T 38d1caa2f7024126b654c4e5ad841e96 P dc70b50a5c82417b8c45a9c453021db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26f5b14b786047d1a6b008e650d7872c in term 0.
I20260806 10:53:53.250293 4390 leader_election.cc:288] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc70b50a5c82417b8c45a9c453021db9 (127.30.218.1:38777), 26f5b14b786047d1a6b008e650d7872c (127.30.218.3:43615)
I20260806 10:53:53.249378 4189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d1caa2f7024126b654c4e5ad841e96" candidate_uuid: "26f5b14b786047d1a6b008e650d7872c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dac00de4c5fe4d7e8ab9209fe9930175" is_pre_election: true
I20260806 10:53:53.251894 4255 leader_election.cc:302] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f5b14b786047d1a6b008e650d7872c, dc70b50a5c82417b8c45a9c453021db9; no voters:
I20260806 10:53:53.251915 4189 raft_consensus.cc:2454] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26f5b14b786047d1a6b008e650d7872c in term 0.
I20260806 10:53:53.252429 4393 raft_consensus.cc:2781] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:53:53.252592 4393 raft_consensus.cc:482] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:53:53.252710 4393 raft_consensus.cc:3037] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:53.253324 4380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:53.245338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:56903 (local address 127.30.218.1:38777)
0806 10:53:53.247027 (+ 1689us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:53.247033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:53.250625 (+ 3592us) server_negotiation.cc:407] Connection header received
0806 10:53:53.250898 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:53.250904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:53.250991 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:53.251062 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:53.252475 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:53.252481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:53.252486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:53:53.252509 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:53.252554 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:53:53.252570 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:53.252666 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:53.252926 (+ 260us) server_negotiation.cc:299] Negotiation successful
0806 10:53:53.253002 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1574,"thread_start_us":4535,"threads_started":1}
I20260806 10:53:53.254175 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:53.245404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:38777 (local address 127.30.218.2:56903)
0806 10:53:53.250514 (+ 5110us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:53.250534 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:53.250725 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:53.251263 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:53.251269 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:53.251295 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:53.252226 (+ 931us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:53:53.252231 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:53.252251 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:53:53.252306 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:53.252756 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:53.252761 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:53.252765 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 10:53:53.253630 (+ 865us) client_negotiation.cc:240] Negotiation successful
0806 10:53:53.253687 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4970,"thread_start_us":4756,"threads_started":1}
I20260806 10:53:53.254889 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:53.247388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:59801 (local address 127.30.218.3:43615)
0806 10:53:53.247558 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:53.247564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:53.251143 (+ 3579us) server_negotiation.cc:407] Connection header received
0806 10:53:53.251304 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:53.251309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:53.251377 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:53.251451 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:53.254129 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:53.254136 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:53.254143 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:53:53.254163 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:53.254202 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:53:53.254223 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:53.254324 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:53.254548 (+ 224us) server_negotiation.cc:299] Negotiation successful
0806 10:53:53.254622 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 10:53:53.254982 4057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d1caa2f7024126b654c4e5ad841e96" candidate_uuid: "dac00de4c5fe4d7e8ab9209fe9930175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc70b50a5c82417b8c45a9c453021db9" is_pre_election: true
I20260806 10:53:53.255235 4057 raft_consensus.cc:2454] T 38d1caa2f7024126b654c4e5ad841e96 P dc70b50a5c82417b8c45a9c453021db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dac00de4c5fe4d7e8ab9209fe9930175 in term 0.
I20260806 10:53:53.255584 4397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:53.246182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:43615 (local address 127.30.218.2:59801)
0806 10:53:53.251027 (+ 4845us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:53.251042 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:53.251152 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:53.252709 (+ 1557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:53.252714 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:53.252740 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:53:53.253870 (+ 1130us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:53:53.253874 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:53:53.253890 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:53:53.253937 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:53:53.254414 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:53:53.254419 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:53:53.254424 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 10:53:53.255113 (+ 689us) client_negotiation.cc:240] Negotiation successful
0806 10:53:53.255166 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4737,"thread_start_us":4562,"threads_started":1}
I20260806 10:53:53.255975 4124 leader_election.cc:302] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dac00de4c5fe4d7e8ab9209fe9930175, dc70b50a5c82417b8c45a9c453021db9; no voters:
I20260806 10:53:53.256461 4390 raft_consensus.cc:2781] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:53:53.256616 4390 raft_consensus.cc:482] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:53:53.256724 4390 raft_consensus.cc:3037] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:53.256583 4393 raft_consensus.cc:504] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc70b50a5c82417b8c45a9c453021db9" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38777 } } peers { permanent_uuid: "dac00de4c5fe4d7e8ab9209fe9930175" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35191 } } peers { permanent_uuid: "26f5b14b786047d1a6b008e650d7872c" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 43615 } }
I20260806 10:53:53.256764 4320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d1caa2f7024126b654c4e5ad841e96" candidate_uuid: "dac00de4c5fe4d7e8ab9209fe9930175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f5b14b786047d1a6b008e650d7872c" is_pre_election: true
I20260806 10:53:53.257369 4320 raft_consensus.cc:2379] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dac00de4c5fe4d7e8ab9209fe9930175 in current term 1: Already voted for candidate 26f5b14b786047d1a6b008e650d7872c in this term.
I20260806 10:53:53.257779 4057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d1caa2f7024126b654c4e5ad841e96" candidate_uuid: "26f5b14b786047d1a6b008e650d7872c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc70b50a5c82417b8c45a9c453021db9"
I20260806 10:53:53.258002 4057 raft_consensus.cc:3037] T 38d1caa2f7024126b654c4e5ad841e96 P dc70b50a5c82417b8c45a9c453021db9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:53.259814 4390 raft_consensus.cc:504] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc70b50a5c82417b8c45a9c453021db9" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38777 } } peers { permanent_uuid: "dac00de4c5fe4d7e8ab9209fe9930175" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35191 } } peers { permanent_uuid: "26f5b14b786047d1a6b008e650d7872c" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 43615 } }
I20260806 10:53:53.260471 4390 leader_election.cc:288] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175 [CANDIDATE]: Term 1 election: Requested vote from peers dc70b50a5c82417b8c45a9c453021db9 (127.30.218.1:38777), 26f5b14b786047d1a6b008e650d7872c (127.30.218.3:43615)
I20260806 10:53:53.260929 4056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d1caa2f7024126b654c4e5ad841e96" candidate_uuid: "dac00de4c5fe4d7e8ab9209fe9930175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc70b50a5c82417b8c45a9c453021db9"
I20260806 10:53:53.261389 4320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d1caa2f7024126b654c4e5ad841e96" candidate_uuid: "dac00de4c5fe4d7e8ab9209fe9930175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f5b14b786047d1a6b008e650d7872c"
I20260806 10:53:53.261502 4057 raft_consensus.cc:2454] T 38d1caa2f7024126b654c4e5ad841e96 P dc70b50a5c82417b8c45a9c453021db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f5b14b786047d1a6b008e650d7872c in term 1.
I20260806 10:53:53.261664 4320 raft_consensus.cc:2379] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate dac00de4c5fe4d7e8ab9209fe9930175 in current term 1: Already voted for candidate 26f5b14b786047d1a6b008e650d7872c in this term.
I20260806 10:53:53.261950 4255 leader_election.cc:302] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f5b14b786047d1a6b008e650d7872c, dc70b50a5c82417b8c45a9c453021db9; no voters:
I20260806 10:53:53.262115 4123 leader_election.cc:302] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dac00de4c5fe4d7e8ab9209fe9930175; no voters: 26f5b14b786047d1a6b008e650d7872c, dc70b50a5c82417b8c45a9c453021db9
I20260806 10:53:53.262866 4398 raft_consensus.cc:2781] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:53.262804 4189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d1caa2f7024126b654c4e5ad841e96" candidate_uuid: "26f5b14b786047d1a6b008e650d7872c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dac00de4c5fe4d7e8ab9209fe9930175"
I20260806 10:53:53.262902 4390 raft_consensus.cc:2726] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 10:53:53.263180 4189 raft_consensus.cc:2379] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 26f5b14b786047d1a6b008e650d7872c in current term 1: Already voted for candidate dac00de4c5fe4d7e8ab9209fe9930175 in this term.
I20260806 10:53:53.263150 4393 leader_election.cc:288] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c [CANDIDATE]: Term 1 election: Requested vote from peers dc70b50a5c82417b8c45a9c453021db9 (127.30.218.1:38777), dac00de4c5fe4d7e8ab9209fe9930175 (127.30.218.2:35191)
I20260806 10:53:53.263983 4398 raft_consensus.cc:686] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c [term 1 LEADER]: Becoming Leader. State: Replica: 26f5b14b786047d1a6b008e650d7872c, State: Running, Role: LEADER
I20260806 10:53:53.264602 4398 consensus_queue.cc:237] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc70b50a5c82417b8c45a9c453021db9" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38777 } } peers { permanent_uuid: "dac00de4c5fe4d7e8ab9209fe9930175" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35191 } } peers { permanent_uuid: "26f5b14b786047d1a6b008e650d7872c" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 43615 } }
I20260806 10:53:53.270102 3915 catalog_manager.cc:5495] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c reported cstate change: term changed from 0 to 1, leader changed from <none> to 26f5b14b786047d1a6b008e650d7872c (127.30.218.3). New cstate: current_term: 1 leader_uuid: "26f5b14b786047d1a6b008e650d7872c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc70b50a5c82417b8c45a9c453021db9" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dac00de4c5fe4d7e8ab9209fe9930175" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f5b14b786047d1a6b008e650d7872c" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 43615 } health_report { overall_health: HEALTHY } } }
I20260806 10:53:53.328159 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:53.325398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40144 (local address 127.30.218.3:43615)
0806 10:53:53.325657 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:53.325664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:53.325842 (+ 178us) server_negotiation.cc:407] Connection header received
0806 10:53:53.326147 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:53.326151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:53.326275 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:53.326387 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:53.327445 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:53.327449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:53.327452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:53:53.327469 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:53.327509 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:53:53.327523 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:53.327642 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:53.327754 (+ 112us) server_negotiation.cc:299] Negotiation successful
0806 10:53:53.327822 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 10:53:53.338256 4057 raft_consensus.cc:1261] T 38d1caa2f7024126b654c4e5ad841e96 P dc70b50a5c82417b8c45a9c453021db9 [term 1 FOLLOWER]: Refusing update from remote peer 26f5b14b786047d1a6b008e650d7872c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:53:53.338281 4189 raft_consensus.cc:1261] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175 [term 1 FOLLOWER]: Refusing update from remote peer 26f5b14b786047d1a6b008e650d7872c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:53:53.339089 4398 consensus_queue.cc:1035] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dac00de4c5fe4d7e8ab9209fe9930175" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 35191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:53:53.339447 4399 consensus_queue.cc:1035] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc70b50a5c82417b8c45a9c453021db9" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:53:53.356980 4405 mvcc.cc:205] Tried to move back new op lower bound from 7315511842141835264 to 7315511841856430080. Current Snapshot: MvccSnapshot[applied={T|T < 7315511842141835264}]
I20260806 10:53:53.369062 3914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:53:53.369503 3914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:53:53.373973 3914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abd22a5c3e5f4a1da4b915194072f733: Sending DeleteTablet for 3 replicas of tablet 38d1caa2f7024126b654c4e5ad841e96
I20260806 10:53:53.375480 4037 tablet_service.cc:1514] Processing DeleteTablet for tablet 38d1caa2f7024126b654c4e5ad841e96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:53:53 UTC) from {username='slave'} at 127.0.0.1:57072
I20260806 10:53:53.375480 4169 tablet_service.cc:1514] Processing DeleteTablet for tablet 38d1caa2f7024126b654c4e5ad841e96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:53:53 UTC) from {username='slave'} at 127.0.0.1:59460
I20260806 10:53:53.375917 4300 tablet_service.cc:1514] Processing DeleteTablet for tablet 38d1caa2f7024126b654c4e5ad841e96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:53:53 UTC) from {username='slave'} at 127.0.0.1:40136
I20260806 10:53:53.381280 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 3974
I20260806 10:53:53.407512 4380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:53.383068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57088 (local address 127.30.218.1:38777)
0806 10:53:53.383343 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:53.383362 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:53.403692 (+ 20330us) server_negotiation.cc:407] Connection header received
0806 10:53:53.404436 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:53.404446 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:53.404589 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:53.404684 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:53.406418 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:53.406425 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:53.406431 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:53:53.406460 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:53.406511 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:53:53.406542 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:53.406661 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:53.407015 (+ 354us) server_negotiation.cc:299] Negotiation successful
0806 10:53:53.407112 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260806 10:53:53.409848 4097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:53.425407 4412 tablet_replica.cc:330] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175: stopping tablet replica
I20260806 10:53:53.425864 4414 tablet_replica.cc:330] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c: stopping tablet replica
I20260806 10:53:53.426177 4412 raft_consensus.cc:2229] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:53:53.426555 4414 raft_consensus.cc:2229] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:53:53.426877 4412 raft_consensus.cc:2258] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:53:53.427831 4414 raft_consensus.cc:2258] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:53:53.429275 4412 ts_tablet_manager.cc:1905] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:53:53.429973 4414 ts_tablet_manager.cc:1905] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:53:53.437536 4414 ts_tablet_manager.cc:1918] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:53:53.437851 4414 log.cc:1196] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/38d1caa2f7024126b654c4e5ad841e96
I20260806 10:53:53.438081 4412 ts_tablet_manager.cc:1918] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:53:53.438284 4412 log.cc:1196] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/38d1caa2f7024126b654c4e5ad841e96
I20260806 10:53:53.438699 4414 ts_tablet_manager.cc:1939] T 38d1caa2f7024126b654c4e5ad841e96 P 26f5b14b786047d1a6b008e650d7872c: Deleting consensus metadata
I20260806 10:53:53.438812 4412 ts_tablet_manager.cc:1939] T 38d1caa2f7024126b654c4e5ad841e96 P dac00de4c5fe4d7e8ab9209fe9930175: Deleting consensus metadata
I20260806 10:53:53.440963 3900 catalog_manager.cc:4841] TS 26f5b14b786047d1a6b008e650d7872c (127.30.218.3:43615): tablet 38d1caa2f7024126b654c4e5ad841e96 (table test-table [id=0cef38fc74224d0fb4abcfdf5a337a89]) successfully deleted
I20260806 10:53:53.441362 3900 catalog_manager.cc:4841] TS dac00de4c5fe4d7e8ab9209fe9930175 (127.30.218.2:35191): tablet 38d1caa2f7024126b654c4e5ad841e96 (table test-table [id=0cef38fc74224d0fb4abcfdf5a337a89]) successfully deleted
I20260806 10:53:53.555310 4413 tablet_replica.cc:330] T 38d1caa2f7024126b654c4e5ad841e96 P dc70b50a5c82417b8c45a9c453021db9: stopping tablet replica
I20260806 10:53:53.555521 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 3974
W20260806 10:53:53.576251 3901 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.218.1:38777 (error 108)
I20260806 10:53:53.577229 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 4106
I20260806 10:53:53.580387 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:53.578742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:38777 (local address 127.0.0.1:57104)
0806 10:53:53.579977 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:53.580112 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1106,"thread_start_us":691,"threads_started":1}
I20260806 10:53:53.580979 4381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:53.578300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59474 (local address 127.30.218.2:35191)
0806 10:53:53.578500 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:53.578505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:53.578630 (+ 125us) server_negotiation.cc:407] Connection header received
0806 10:53:53.578886 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:53.578894 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:53.579043 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:53.579166 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:53.580246 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:53.580254 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:53.580259 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:53:53.580281 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:53.580330 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:53:53.580349 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:53.580499 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:53.580590 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 10:53:53.580661 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260806 10:53:53.580905 3901 catalog_manager.cc:4584] TS dc70b50a5c82417b8c45a9c453021db9 (127.30.218.1:38777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38d1caa2f7024126b654c4e5ad841e96: Network error: Client connection negotiation failed: client connection to 127.30.218.1:38777: connect: Connection refused (error 111)
I20260806 10:53:53.582085 4229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:53.635406 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:53.634657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:38777 (local address 127.0.0.1:57108)
0806 10:53:53.635040 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:53.635178 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215}
I20260806 10:53:53.637006 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:53.636481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:38777 (local address 127.0.0.1:57112)
0806 10:53:53.636705 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:53.636808 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260806 10:53:53.685243 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 4106
I20260806 10:53:53.697898 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:53.697185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:38777 (local address 127.0.0.1:57118)
0806 10:53:53.697491 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:53.697629 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260806 10:53:53.699209 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:53.698772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:38777 (local address 127.0.0.1:57126)
0806 10:53:53.698941 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:53.699030 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 10:53:53.713532 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 4237
I20260806 10:53:53.716933 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:53.714582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40150 (local address 127.30.218.3:43615)
0806 10:53:53.714759 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:53.714764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:53.714784 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:53:53.715025 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:53.715030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:53.715093 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:53.715184 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:53.716237 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:53.716243 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:53.716247 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:53:53.716264 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:53.716301 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:53:53.716313 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:53.716431 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:53.716552 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 10:53:53.716608 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 10:53:53.717883 4360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:53.780336 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:53.779686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:38777 (local address 127.0.0.1:57142)
0806 10:53:53.780017 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:53.780142 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260806 10:53:53.781713 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:53.781155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:38777 (local address 127.0.0.1:57154)
0806 10:53:53.781402 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:53.781542 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260806 10:53:53.856346 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 4237
I20260806 10:53:53.880271 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 3883
I20260806 10:53:53.881788 3945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:54.011865 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 3883
2026-08-06T10:53:54Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3301 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 10:53:54.061185 31592 mini_oidc.cc:152] Starting JWKS server
I20260806 10:53:54.064937 31592 webserver.cc:460] Webserver started at https://127.0.0.1:45895/ using document root <none> and password file <none>
I20260806 10:53:54.065330 31592 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 10:53:54.066166 31592 webserver.cc:460] Webserver started at http://127.0.0.1:33119/ using document root <none> and password file <none>
2026-08-06T10:53:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:53:54Z Disabled control of system clock
I20260806 10:53:54.112468 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:40821
--webserver_interface=127.30.218.62
--webserver_port=0
--builtin_ntp_servers=127.30.218.20:34469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:40821
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45895/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013589554914-31592-0/testchainca.cert with env {}
W20260806 10:53:54.471275 4427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:54.471606 4427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:54.471726 4427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:54.479189 4427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:53:54.479321 4427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:54.479383 4427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:54.479462 4427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:53:54.479516 4427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 10:53:54.479602 4427 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 10:53:54.479648 4427 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45895/jwks.json
I20260806 10:53:54.489982 4427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:34469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:40821
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:40821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45895/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013589554914-31592-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:54.491390 4427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:54.492679 4427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:54.504307 4432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:54.504350 4433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:54.504436 4435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:54.505353 4427 server_base.cc:1034] running on GCE node
I20260806 10:53:54.506175 4427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:54.507620 4427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:54.508821 4427 hybrid_clock.cc:648] HybridClock initialized: now 1786013634508789 us; error 34 us; skew 500 ppm
I20260806 10:53:54.509176 4427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:53:54.514747 4427 webserver.cc:460] Webserver started at http://127.30.218.62:39605/ using document root <none> and password file <none>
I20260806 10:53:54.515448 4427 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:54.515543 4427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:54.515815 4427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:54.518877 4427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "dd5354f508604051a6bcf177611ba812"
format_stamp: "Formatted at 2026-08-06 10:53:54 on dist-test-slave-49h3"
server_key: "58a79f44d897912635bcf5660c75339b"
server_key_iv: "2f11b90457e19e9c4ee07816d3b720fc"
server_key_version: "encryptionkey@0"
I20260806 10:53:54.519665 4427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "dd5354f508604051a6bcf177611ba812"
format_stamp: "Formatted at 2026-08-06 10:53:54 on dist-test-slave-49h3"
server_key: "58a79f44d897912635bcf5660c75339b"
server_key_iv: "2f11b90457e19e9c4ee07816d3b720fc"
server_key_version: "encryptionkey@0"
I20260806 10:53:54.524720 4427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 10:53:54.528132 4441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:54.529424 4427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:53:54.529630 4427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "dd5354f508604051a6bcf177611ba812"
format_stamp: "Formatted at 2026-08-06 10:53:54 on dist-test-slave-49h3"
server_key: "58a79f44d897912635bcf5660c75339b"
server_key_iv: "2f11b90457e19e9c4ee07816d3b720fc"
server_key_version: "encryptionkey@0"
I20260806 10:53:54.529831 4427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:54.547739 4427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:54.548714 4427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:54.549037 4427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:54.568173 4427 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:40821
I20260806 10:53:54.568187 4493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:40821 every 8 connection(s)
I20260806 10:53:54.569674 4427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:53:54.572335 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 4427
I20260806 10:53:54.572561 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:53:54.573413 31592 external_mini_cluster.cc:1428] Setting key 728db56ef2bdbb0c1f96df4c265f19b1
I20260806 10:53:54.574151 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:54.587580 4494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd5354f508604051a6bcf177611ba812: Bootstrap starting.
I20260806 10:53:54.588203 4497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:54.575528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34094 (local address 127.30.218.62:40821)
0806 10:53:54.576515 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:54.576526 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:54.576567 (+ 41us) server_negotiation.cc:407] Connection header received
0806 10:53:54.578890 (+ 2323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:54.578914 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:54.579407 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:54.579809 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:54.580435 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:54.581879 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:54.583332 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:54.584069 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:54.584272 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:54.585634 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:54.585643 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:54.585661 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:53:54.586269 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:54.586362 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:53:54.586386 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:54.586507 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:54.586757 (+ 250us) server_negotiation.cc:299] Negotiation successful
0806 10:53:54.587112 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":359,"threads_started":1}
I20260806 10:53:54.591199 4494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd5354f508604051a6bcf177611ba812: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:54.592407 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:53:54.592392 4494 log.cc:824] T 00000000000000000000000000000000 P dd5354f508604051a6bcf177611ba812: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:54.595239 4494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd5354f508604051a6bcf177611ba812: No bootstrap required, opened a new log
I20260806 10:53:54.602794 4494 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd5354f508604051a6bcf177611ba812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd5354f508604051a6bcf177611ba812" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40821 } }
I20260806 10:53:54.603143 4494 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd5354f508604051a6bcf177611ba812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:54.603245 4494 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd5354f508604051a6bcf177611ba812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd5354f508604051a6bcf177611ba812, State: Initialized, Role: FOLLOWER
I20260806 10:53:54.603824 4494 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd5354f508604051a6bcf177611ba812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd5354f508604051a6bcf177611ba812" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40821 } }
I20260806 10:53:54.604054 4494 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd5354f508604051a6bcf177611ba812 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:53:54.604178 4494 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd5354f508604051a6bcf177611ba812 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:53:54.604321 4494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd5354f508604051a6bcf177611ba812 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:54.606802 4494 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd5354f508604051a6bcf177611ba812 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd5354f508604051a6bcf177611ba812" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40821 } }
I20260806 10:53:54.607281 4494 leader_election.cc:302] T 00000000000000000000000000000000 P dd5354f508604051a6bcf177611ba812 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd5354f508604051a6bcf177611ba812; no voters:
I20260806 10:53:54.607851 4494 leader_election.cc:288] T 00000000000000000000000000000000 P dd5354f508604051a6bcf177611ba812 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:53:54.607990 4499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd5354f508604051a6bcf177611ba812 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:54.608690 4499 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd5354f508604051a6bcf177611ba812 [term 1 LEADER]: Becoming Leader. State: Replica: dd5354f508604051a6bcf177611ba812, State: Running, Role: LEADER
I20260806 10:53:54.609172 4499 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd5354f508604051a6bcf177611ba812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd5354f508604051a6bcf177611ba812" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40821 } }
I20260806 10:53:54.609894 4494 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd5354f508604051a6bcf177611ba812 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:53:54.611965 4501 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd5354f508604051a6bcf177611ba812 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd5354f508604051a6bcf177611ba812. Latest consensus state: current_term: 1 leader_uuid: "dd5354f508604051a6bcf177611ba812" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd5354f508604051a6bcf177611ba812" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40821 } } }
I20260806 10:53:54.612407 4501 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd5354f508604051a6bcf177611ba812 [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:54.611938 4500 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd5354f508604051a6bcf177611ba812 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd5354f508604051a6bcf177611ba812" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd5354f508604051a6bcf177611ba812" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40821 } } }
I20260806 10:53:54.613125 4500 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd5354f508604051a6bcf177611ba812 [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:54.618731 4507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:53:54.623636 4507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:53:54.632357 4507 catalog_manager.cc:1347] Generated new cluster ID: 4794e39ce379480daeb3b1936f54b1ae
I20260806 10:53:54.632507 4507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:53:54.655495 4507 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:53:54.657428 4507 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:53:54.668082 4507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd5354f508604051a6bcf177611ba812: Generated new TSK 0
I20260806 10:53:54.669319 4507 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 10:53:54.722572 4524 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 10:53:54.722832 4524 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 10:53:54.723877 4497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:54.694522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34096 (local address 127.30.218.62:40821)
0806 10:53:54.694731 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:54.694737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:54.695151 (+ 414us) server_negotiation.cc:407] Connection header received
0806 10:53:54.695509 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:54.695516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:54.695619 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:54.695791 (+ 172us) server_negotiation.cc:226] Negotiated authn=JWT
0806 10:53:54.696305 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:54.697495 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:54.698975 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:54.699452 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:54.699586 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:54.700122 (+ 536us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 10:53:54.723083 (+ 22961us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 10:53:54.723513 (+ 430us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34096: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":77}
W20260806 10:53:54.724030 4497 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34096: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 10:53:54.725786 4523 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:53:54.694370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:40821 (local address 127.0.0.1:34096)
0806 10:53:54.695040 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:54.695059 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:54.695221 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:54.695806 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:54.695809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:54.695820 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0806 10:53:54.696149 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:54.696157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:54.697661 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:54.697665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:54.698816 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:54.698829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:54.699628 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:54.699635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:54.699774 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:54.699843 (+ 69us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 10:53:54.723449 (+ 23606us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 10:53:54.723856 (+ 407us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.218.62:40821: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":338,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.30.218.62:40821: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 10:53:54.728031 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 4427
I20260806 10:53:54.729382 4489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:54.827402 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 4427
2026-08-06T10:53:54Z chronyd exiting
I20260806 10:53:54.879371 31592 test_util.cc:175] -----------------------------------------------
I20260806 10:53:54.879506 31592 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786013589554914-31592-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (823 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 10:53:54.883556 31592 mini_oidc.cc:152] Starting JWKS server
I20260806 10:53:54.885488 31592 webserver.cc:460] Webserver started at https://127.0.0.1:38653/ using document root <none> and password file <none>
I20260806 10:53:54.885768 31592 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 10:53:54.886420 31592 webserver.cc:460] Webserver started at http://127.0.0.1:40535/ using document root <none> and password file <none>
2026-08-06T10:53:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:53:54Z Disabled control of system clock
I20260806 10:53:54.925154 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:40769
--webserver_interface=127.30.218.62
--webserver_port=0
--builtin_ntp_servers=127.30.218.20:45161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:40769
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38653/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013589554914-31592-0/testchainca.cert with env {}
W20260806 10:53:55.263249 4531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:55.263552 4531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:55.263684 4531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:55.271169 4531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:53:55.271286 4531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:55.271353 4531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:55.271435 4531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:53:55.271497 4531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 10:53:55.271582 4531 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 10:53:55.271636 4531 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38653/jwks.json
I20260806 10:53:55.281669 4531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:45161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:40769
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:40769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38653/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013589554914-31592-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:55.282927 4531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:55.284192 4531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:55.296001 4536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:55.296070 4537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:55.296002 4539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:55.296940 4531 server_base.cc:1034] running on GCE node
I20260806 10:53:55.297780 4531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:55.299250 4531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:55.300462 4531 hybrid_clock.cc:648] HybridClock initialized: now 1786013635300411 us; error 48 us; skew 500 ppm
I20260806 10:53:55.300812 4531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:53:55.303141 4531 webserver.cc:460] Webserver started at http://127.30.218.62:41243/ using document root <none> and password file <none>
I20260806 10:53:55.303843 4531 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:55.303956 4531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:55.304276 4531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:55.307250 4531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "f8239b71b8b74f658790ff041dde651a"
format_stamp: "Formatted at 2026-08-06 10:53:55 on dist-test-slave-49h3"
server_key: "71e1db84329d55fc239e202556910ce7"
server_key_iv: "8382ee4790c5fb54ebe3d1956f78cc44"
server_key_version: "encryptionkey@0"
I20260806 10:53:55.308049 4531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "f8239b71b8b74f658790ff041dde651a"
format_stamp: "Formatted at 2026-08-06 10:53:55 on dist-test-slave-49h3"
server_key: "71e1db84329d55fc239e202556910ce7"
server_key_iv: "8382ee4790c5fb54ebe3d1956f78cc44"
server_key_version: "encryptionkey@0"
I20260806 10:53:55.313567 4531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260806 10:53:55.317243 4545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:55.318794 4531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 10:53:55.319002 4531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "f8239b71b8b74f658790ff041dde651a"
format_stamp: "Formatted at 2026-08-06 10:53:55 on dist-test-slave-49h3"
server_key: "71e1db84329d55fc239e202556910ce7"
server_key_iv: "8382ee4790c5fb54ebe3d1956f78cc44"
server_key_version: "encryptionkey@0"
I20260806 10:53:55.319240 4531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:55.348953 4531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:55.349962 4531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:55.350299 4531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:55.368991 4531 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:40769
I20260806 10:53:55.368999 4597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:40769 every 8 connection(s)
I20260806 10:53:55.370426 4531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:53:55.371161 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 4531
I20260806 10:53:55.371378 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:53:55.372164 31592 external_mini_cluster.cc:1428] Setting key 5bcbf1ae18b77fd609b40a0f7cbb26cd
I20260806 10:53:55.374943 4598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:55.386116 4601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:55.374204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49204 (local address 127.30.218.62:40769)
0806 10:53:55.375291 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:55.375303 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:55.375347 (+ 44us) server_negotiation.cc:407] Connection header received
0806 10:53:55.377015 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:55.377038 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:55.377605 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:55.378003 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:55.378601 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:55.379949 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:55.381277 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:55.382006 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:55.382195 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:55.383487 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:55.383498 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:55.383515 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:53:55.384169 (+ 654us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:55.384254 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:53:55.384285 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:55.384394 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:55.384632 (+ 238us) server_negotiation.cc:299] Negotiation successful
0806 10:53:55.384987 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":379,"threads_started":1}
I20260806 10:53:55.386415 4598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8239b71b8b74f658790ff041dde651a: Bootstrap starting.
I20260806 10:53:55.390265 4598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8239b71b8b74f658790ff041dde651a: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:55.390302 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:53:55.391319 4598 log.cc:824] T 00000000000000000000000000000000 P f8239b71b8b74f658790ff041dde651a: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:55.394637 4598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8239b71b8b74f658790ff041dde651a: No bootstrap required, opened a new log
I20260806 10:53:55.402647 4598 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8239b71b8b74f658790ff041dde651a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8239b71b8b74f658790ff041dde651a" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40769 } }
I20260806 10:53:55.403046 4598 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8239b71b8b74f658790ff041dde651a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:55.403153 4598 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8239b71b8b74f658790ff041dde651a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8239b71b8b74f658790ff041dde651a, State: Initialized, Role: FOLLOWER
I20260806 10:53:55.403719 4598 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8239b71b8b74f658790ff041dde651a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8239b71b8b74f658790ff041dde651a" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40769 } }
I20260806 10:53:55.403949 4598 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8239b71b8b74f658790ff041dde651a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:53:55.404090 4598 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8239b71b8b74f658790ff041dde651a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:53:55.404259 4598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8239b71b8b74f658790ff041dde651a [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:55.406642 4598 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8239b71b8b74f658790ff041dde651a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8239b71b8b74f658790ff041dde651a" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40769 } }
I20260806 10:53:55.407209 4598 leader_election.cc:302] T 00000000000000000000000000000000 P f8239b71b8b74f658790ff041dde651a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8239b71b8b74f658790ff041dde651a; no voters:
I20260806 10:53:55.407809 4598 leader_election.cc:288] T 00000000000000000000000000000000 P f8239b71b8b74f658790ff041dde651a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:53:55.407941 4603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8239b71b8b74f658790ff041dde651a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:55.408632 4603 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8239b71b8b74f658790ff041dde651a [term 1 LEADER]: Becoming Leader. State: Replica: f8239b71b8b74f658790ff041dde651a, State: Running, Role: LEADER
I20260806 10:53:55.409294 4603 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8239b71b8b74f658790ff041dde651a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8239b71b8b74f658790ff041dde651a" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40769 } }
I20260806 10:53:55.410163 4598 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8239b71b8b74f658790ff041dde651a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:53:55.412305 4604 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8239b71b8b74f658790ff041dde651a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8239b71b8b74f658790ff041dde651a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8239b71b8b74f658790ff041dde651a" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40769 } } }
I20260806 10:53:55.412284 4605 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8239b71b8b74f658790ff041dde651a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8239b71b8b74f658790ff041dde651a. Latest consensus state: current_term: 1 leader_uuid: "f8239b71b8b74f658790ff041dde651a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8239b71b8b74f658790ff041dde651a" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40769 } } }
I20260806 10:53:55.412899 4604 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8239b71b8b74f658790ff041dde651a [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:55.412906 4605 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8239b71b8b74f658790ff041dde651a [sys.catalog]: This master's current role is: LEADER
W20260806 10:53:55.421779 4619 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8239b71b8b74f658790ff041dde651a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 10:53:55.421900 4619 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 10:53:55.422029 4610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:53:55.424273 4610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:53:55.434270 4610 catalog_manager.cc:1347] Generated new cluster ID: 30822fa1e8a444f7b8faf8c15f81707c
I20260806 10:53:55.434417 4610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:53:55.454054 4610 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:53:55.455415 4610 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:53:55.473814 4610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8239b71b8b74f658790ff041dde651a: Generated new TSK 0
I20260806 10:53:55.474805 4610 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 10:53:55.516295 4628 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 10:53:55.516467 4628 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 10:53:55.517627 4601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:55.497156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49212 (local address 127.30.218.62:40769)
0806 10:53:55.497373 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:55.497380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:55.497744 (+ 364us) server_negotiation.cc:407] Connection header received
0806 10:53:55.497999 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:55.498005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:55.498139 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:55.498358 (+ 219us) server_negotiation.cc:226] Negotiated authn=JWT
0806 10:53:55.498889 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:55.500036 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:55.501491 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:55.502210 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:55.502347 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:55.502933 (+ 586us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 10:53:55.516900 (+ 13967us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 10:53:55.517268 (+ 368us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49212: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":84}
W20260806 10:53:55.517681 4627 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:53:55.496955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:40769 (local address 127.0.0.1:49212)
0806 10:53:55.497638 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:55.497655 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:55.497802 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:55.498403 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:55.498407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:55.498419 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0806 10:53:55.498706 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:55.498715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:55.500209 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:55.500213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:55.501317 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:55.501330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:55.502387 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:55.502392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:55.502565 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:55.502605 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 10:53:55.517109 (+ 14504us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 10:53:55.517332 (+ 223us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.218.62:40769: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":347,"threads_started":1}
W20260806 10:53:55.517752 4601 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49212: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260806 10:53:55.519142 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 4531
I20260806 10:53:55.520222 4593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:55.617084 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 4531
2026-08-06T10:53:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (789 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 10:53:55.672688 31592 mini_oidc.cc:152] Starting JWKS server
I20260806 10:53:55.674690 31592 webserver.cc:460] Webserver started at https://127.0.0.1:46819/ using document root <none> and password file <none>
I20260806 10:53:55.674925 31592 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 10:53:55.675647 31592 webserver.cc:460] Webserver started at http://127.0.0.1:44935/ using document root <none> and password file <none>
2026-08-06T10:53:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:53:55Z Disabled control of system clock
I20260806 10:53:55.710592 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:45975
--webserver_interface=127.30.218.62
--webserver_port=0
--builtin_ntp_servers=127.30.218.20:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:45975
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46819/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 10:53:56.047940 4635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:56.048313 4635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:56.048456 4635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:56.056048 4635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:53:56.056213 4635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:56.056284 4635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:56.056368 4635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:53:56.056427 4635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 10:53:56.056514 4635 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 10:53:56.056566 4635 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46819/jwks.json
I20260806 10:53:56.066413 4635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:45975
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:45975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46819/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:56.067809 4635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:56.069718 4635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:56.081285 4641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:56.081372 4642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:56.081318 4644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:56.082399 4635 server_base.cc:1034] running on GCE node
I20260806 10:53:56.083145 4635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:56.084517 4635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:56.085717 4635 hybrid_clock.cc:648] HybridClock initialized: now 1786013636085680 us; error 32 us; skew 500 ppm
I20260806 10:53:56.086074 4635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:53:56.088272 4635 webserver.cc:460] Webserver started at http://127.30.218.62:36965/ using document root <none> and password file <none>
I20260806 10:53:56.088949 4635 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:56.089057 4635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:56.089357 4635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:56.092254 4635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "0020b88375484b929e6b3e44d0c51b86"
format_stamp: "Formatted at 2026-08-06 10:53:56 on dist-test-slave-49h3"
server_key: "75726fb12379f297a6cdd9dc1357f2df"
server_key_iv: "5679d0b7a02c647023a69f8d0dab3775"
server_key_version: "encryptionkey@0"
I20260806 10:53:56.093021 4635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "0020b88375484b929e6b3e44d0c51b86"
format_stamp: "Formatted at 2026-08-06 10:53:56 on dist-test-slave-49h3"
server_key: "75726fb12379f297a6cdd9dc1357f2df"
server_key_iv: "5679d0b7a02c647023a69f8d0dab3775"
server_key_version: "encryptionkey@0"
I20260806 10:53:56.097944 4635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 10:53:56.101277 4650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:56.102577 4635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:53:56.102737 4635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "0020b88375484b929e6b3e44d0c51b86"
format_stamp: "Formatted at 2026-08-06 10:53:56 on dist-test-slave-49h3"
server_key: "75726fb12379f297a6cdd9dc1357f2df"
server_key_iv: "5679d0b7a02c647023a69f8d0dab3775"
server_key_version: "encryptionkey@0"
I20260806 10:53:56.102933 4635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:56.116339 4635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:56.117182 4635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:56.117547 4635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:56.135164 4635 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:45975
I20260806 10:53:56.135200 4702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:45975 every 8 connection(s)
I20260806 10:53:56.136534 4635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:53:56.140472 4703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:56.144927 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 4635
I20260806 10:53:56.145146 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:53:56.146049 31592 external_mini_cluster.cc:1428] Setting key 5f58459b0953d8bd8ce7f3f6397dd8f5
I20260806 10:53:56.152616 4703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0020b88375484b929e6b3e44d0c51b86: Bootstrap starting.
I20260806 10:53:56.155725 4703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0020b88375484b929e6b3e44d0c51b86: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:56.156769 4703 log.cc:824] T 00000000000000000000000000000000 P 0020b88375484b929e6b3e44d0c51b86: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:56.160141 4706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:56.148143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49336 (local address 127.30.218.62:45975)
0806 10:53:56.149185 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:56.149196 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:56.149243 (+ 47us) server_negotiation.cc:407] Connection header received
0806 10:53:56.151253 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:56.151279 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:56.151772 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:56.152140 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:56.152698 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:56.154106 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:56.155442 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:56.156181 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:56.156345 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:56.157582 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:56.157590 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:56.157604 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:53:56.158163 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:56.158273 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:53:56.158299 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:56.158426 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:56.158614 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 10:53:56.158934 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":373,"threads_started":1}
I20260806 10:53:56.160198 4703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0020b88375484b929e6b3e44d0c51b86: No bootstrap required, opened a new log
I20260806 10:53:56.164156 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:53:56.169790 4703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0020b88375484b929e6b3e44d0c51b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0020b88375484b929e6b3e44d0c51b86" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 45975 } }
I20260806 10:53:56.170176 4703 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0020b88375484b929e6b3e44d0c51b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:56.170260 4703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0020b88375484b929e6b3e44d0c51b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0020b88375484b929e6b3e44d0c51b86, State: Initialized, Role: FOLLOWER
I20260806 10:53:56.170789 4703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0020b88375484b929e6b3e44d0c51b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0020b88375484b929e6b3e44d0c51b86" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 45975 } }
I20260806 10:53:56.171001 4703 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0020b88375484b929e6b3e44d0c51b86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:53:56.171108 4703 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0020b88375484b929e6b3e44d0c51b86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:53:56.171240 4703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0020b88375484b929e6b3e44d0c51b86 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:56.173640 4703 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0020b88375484b929e6b3e44d0c51b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0020b88375484b929e6b3e44d0c51b86" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 45975 } }
I20260806 10:53:56.174124 4703 leader_election.cc:302] T 00000000000000000000000000000000 P 0020b88375484b929e6b3e44d0c51b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0020b88375484b929e6b3e44d0c51b86; no voters:
I20260806 10:53:56.174695 4703 leader_election.cc:288] T 00000000000000000000000000000000 P 0020b88375484b929e6b3e44d0c51b86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:53:56.174834 4708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0020b88375484b929e6b3e44d0c51b86 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:56.175546 4708 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0020b88375484b929e6b3e44d0c51b86 [term 1 LEADER]: Becoming Leader. State: Replica: 0020b88375484b929e6b3e44d0c51b86, State: Running, Role: LEADER
I20260806 10:53:56.176067 4708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0020b88375484b929e6b3e44d0c51b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0020b88375484b929e6b3e44d0c51b86" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 45975 } }
I20260806 10:53:56.176820 4703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0020b88375484b929e6b3e44d0c51b86 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:53:56.178790 4710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0020b88375484b929e6b3e44d0c51b86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0020b88375484b929e6b3e44d0c51b86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0020b88375484b929e6b3e44d0c51b86" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 45975 } } }
I20260806 10:53:56.178944 4711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0020b88375484b929e6b3e44d0c51b86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0020b88375484b929e6b3e44d0c51b86. Latest consensus state: current_term: 1 leader_uuid: "0020b88375484b929e6b3e44d0c51b86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0020b88375484b929e6b3e44d0c51b86" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 45975 } } }
I20260806 10:53:56.179275 4710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0020b88375484b929e6b3e44d0c51b86 [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:56.179281 4711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0020b88375484b929e6b3e44d0c51b86 [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:56.180148 4714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:53:56.186336 4714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:53:56.196123 4714 catalog_manager.cc:1347] Generated new cluster ID: 66ce426f7f5a4f059956fede54be2700
I20260806 10:53:56.196270 4714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:53:56.211596 4714 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:53:56.213608 4714 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:53:56.235507 4714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0020b88375484b929e6b3e44d0c51b86: Generated new TSK 0
I20260806 10:53:56.236724 4714 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:53:56.258898 4706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:56.250922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49346 (local address 127.30.218.62:45975)
0806 10:53:56.251130 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:56.251135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:56.251484 (+ 349us) server_negotiation.cc:407] Connection header received
0806 10:53:56.251706 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:56.251710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:56.251781 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:56.251909 (+ 128us) server_negotiation.cc:226] Negotiated authn=JWT
0806 10:53:56.252427 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:56.253136 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:56.255178 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:56.255646 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:56.255747 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:56.256357 (+ 610us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 10:53:56.258240 (+ 1883us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46819/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 10:53:56.258579 (+ 339us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49346: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46819/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":83}
W20260806 10:53:56.259040 4706 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49346: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46819/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 10:53:56.259130 4733 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:53:56.250722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:45975 (local address 127.0.0.1:49346)
0806 10:53:56.251377 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:56.251398 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:56.251551 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:56.251945 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:56.251950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:56.251963 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0806 10:53:56.252282 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:56.252294 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:56.253278 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:56.253284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:56.255012 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:56.255029 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:56.255850 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:56.255858 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:56.256050 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:56.256098 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 10:53:56.258444 (+ 2346us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46819/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 10:53:56.258699 (+ 255us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.218.62:45975: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46819/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":327,"threads_started":1}
I20260806 10:53:56.260630 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 4635
I20260806 10:53:56.261722 4698 generic_service.cc:196] Checking for leaks (request via RPC)
E20260806 10:53:56.270082 4734 webserver.cc:533] Webserver: sslize failed:
E20260806 10:53:56.270213 4734 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 10:53:56.363888 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 4635
2026-08-06T10:53:56Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (747 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T10:53:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:53:56Z Disabled control of system clock
I20260806 10:53:56.458308 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:42567
--webserver_interface=127.30.218.62
--webserver_port=0
--builtin_ntp_servers=127.30.218.20:37881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:42567
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013589554914-31592-0/testchainca.cert with env {}
W20260806 10:53:56.800858 4739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:56.801218 4739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:56.801314 4739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:56.808830 4739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:53:56.808946 4739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:56.808984 4739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:56.809041 4739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:53:56.809085 4739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 10:53:56.809152 4739 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 10:53:56.809198 4739 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 10:53:56.819499 4739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:37881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:42567
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:42567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013589554914-31592-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:56.820783 4739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:56.822079 4739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:56.834317 4744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:56.834363 4745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:56.834280 4747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:56.835232 4739 server_base.cc:1034] running on GCE node
I20260806 10:53:56.835994 4739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:56.837570 4739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:56.838770 4739 hybrid_clock.cc:648] HybridClock initialized: now 1786013636838730 us; error 35 us; skew 500 ppm
I20260806 10:53:56.839100 4739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:53:56.841274 4739 webserver.cc:460] Webserver started at http://127.30.218.62:35077/ using document root <none> and password file <none>
I20260806 10:53:56.841969 4739 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:56.842060 4739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:56.842322 4739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:56.845131 4739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "e37d0e43e5d8407ca8bfc05d45d2265a"
format_stamp: "Formatted at 2026-08-06 10:53:56 on dist-test-slave-49h3"
server_key: "91bc3a35438cc7e2e4416788d4589ec1"
server_key_iv: "355b99969838d5314eb0e9348e4ed30a"
server_key_version: "encryptionkey@0"
I20260806 10:53:56.845933 4739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "e37d0e43e5d8407ca8bfc05d45d2265a"
format_stamp: "Formatted at 2026-08-06 10:53:56 on dist-test-slave-49h3"
server_key: "91bc3a35438cc7e2e4416788d4589ec1"
server_key_iv: "355b99969838d5314eb0e9348e4ed30a"
server_key_version: "encryptionkey@0"
I20260806 10:53:56.850942 4739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 10:53:56.854516 4753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:56.856114 4739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:53:56.856284 4739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "e37d0e43e5d8407ca8bfc05d45d2265a"
format_stamp: "Formatted at 2026-08-06 10:53:56 on dist-test-slave-49h3"
server_key: "91bc3a35438cc7e2e4416788d4589ec1"
server_key_iv: "355b99969838d5314eb0e9348e4ed30a"
server_key_version: "encryptionkey@0"
I20260806 10:53:56.856488 4739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:56.871629 4739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:56.872485 4739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:56.872750 4739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:56.891471 4739 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:42567
I20260806 10:53:56.891490 4804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:42567 every 8 connection(s)
I20260806 10:53:56.892997 4739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:53:56.894613 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 4739
I20260806 10:53:56.894879 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:53:56.895722 31592 external_mini_cluster.cc:1428] Setting key bb96101f69a6edc8ce6b4da2fe72b4eb
I20260806 10:53:56.898239 4805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:56.910760 4808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:56.897850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60514 (local address 127.30.218.62:42567)
0806 10:53:56.899421 (+ 1571us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:56.899437 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:56.899493 (+ 56us) server_negotiation.cc:407] Connection header received
0806 10:53:56.901431 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:56.901491 (+ 60us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:56.902056 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:56.902471 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:56.903086 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:56.904513 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:56.905942 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:56.906692 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:56.906877 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:56.908211 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:56.908221 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:56.908238 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:53:56.908868 (+ 630us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:56.908965 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:53:56.908988 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:56.909098 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:56.909344 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 10:53:56.909718 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851,"thread_start_us":503,"threads_started":1}
I20260806 10:53:56.911163 4805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e37d0e43e5d8407ca8bfc05d45d2265a: Bootstrap starting.
I20260806 10:53:56.915017 4805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e37d0e43e5d8407ca8bfc05d45d2265a: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:56.915093 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:53:56.916105 4805 log.cc:824] T 00000000000000000000000000000000 P e37d0e43e5d8407ca8bfc05d45d2265a: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:56.919559 4805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e37d0e43e5d8407ca8bfc05d45d2265a: No bootstrap required, opened a new log
I20260806 10:53:56.927817 4805 raft_consensus.cc:348] T 00000000000000000000000000000000 P e37d0e43e5d8407ca8bfc05d45d2265a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37d0e43e5d8407ca8bfc05d45d2265a" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42567 } }
I20260806 10:53:56.928268 4805 raft_consensus.cc:374] T 00000000000000000000000000000000 P e37d0e43e5d8407ca8bfc05d45d2265a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:56.928364 4805 raft_consensus.cc:729] T 00000000000000000000000000000000 P e37d0e43e5d8407ca8bfc05d45d2265a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e37d0e43e5d8407ca8bfc05d45d2265a, State: Initialized, Role: FOLLOWER
I20260806 10:53:56.928972 4805 consensus_queue.cc:260] T 00000000000000000000000000000000 P e37d0e43e5d8407ca8bfc05d45d2265a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37d0e43e5d8407ca8bfc05d45d2265a" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42567 } }
I20260806 10:53:56.929225 4805 raft_consensus.cc:388] T 00000000000000000000000000000000 P e37d0e43e5d8407ca8bfc05d45d2265a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:53:56.929351 4805 raft_consensus.cc:482] T 00000000000000000000000000000000 P e37d0e43e5d8407ca8bfc05d45d2265a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:53:56.929563 4805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e37d0e43e5d8407ca8bfc05d45d2265a [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:56.931880 4805 raft_consensus.cc:504] T 00000000000000000000000000000000 P e37d0e43e5d8407ca8bfc05d45d2265a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37d0e43e5d8407ca8bfc05d45d2265a" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42567 } }
I20260806 10:53:56.932466 4805 leader_election.cc:302] T 00000000000000000000000000000000 P e37d0e43e5d8407ca8bfc05d45d2265a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e37d0e43e5d8407ca8bfc05d45d2265a; no voters:
I20260806 10:53:56.933085 4805 leader_election.cc:288] T 00000000000000000000000000000000 P e37d0e43e5d8407ca8bfc05d45d2265a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:53:56.933246 4810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e37d0e43e5d8407ca8bfc05d45d2265a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:56.935374 4805 sys_catalog.cc:564] T 00000000000000000000000000000000 P e37d0e43e5d8407ca8bfc05d45d2265a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:53:56.940218 4810 raft_consensus.cc:686] T 00000000000000000000000000000000 P e37d0e43e5d8407ca8bfc05d45d2265a [term 1 LEADER]: Becoming Leader. State: Replica: e37d0e43e5d8407ca8bfc05d45d2265a, State: Running, Role: LEADER
I20260806 10:53:56.940800 4810 consensus_queue.cc:237] T 00000000000000000000000000000000 P e37d0e43e5d8407ca8bfc05d45d2265a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37d0e43e5d8407ca8bfc05d45d2265a" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42567 } }
I20260806 10:53:56.943749 4812 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37d0e43e5d8407ca8bfc05d45d2265a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e37d0e43e5d8407ca8bfc05d45d2265a. Latest consensus state: current_term: 1 leader_uuid: "e37d0e43e5d8407ca8bfc05d45d2265a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37d0e43e5d8407ca8bfc05d45d2265a" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42567 } } }
I20260806 10:53:56.944703 4811 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37d0e43e5d8407ca8bfc05d45d2265a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e37d0e43e5d8407ca8bfc05d45d2265a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37d0e43e5d8407ca8bfc05d45d2265a" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42567 } } }
I20260806 10:53:56.944993 4811 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37d0e43e5d8407ca8bfc05d45d2265a [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:56.944275 4812 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37d0e43e5d8407ca8bfc05d45d2265a [sys.catalog]: This master's current role is: LEADER
W20260806 10:53:56.949177 4824 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e37d0e43e5d8407ca8bfc05d45d2265a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 10:53:56.949317 4824 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 10:53:56.949424 4826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:53:56.951117 4826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:53:56.960500 4826 catalog_manager.cc:1347] Generated new cluster ID: 9f3bb9b473c54fda9fbe6898b5fee358
I20260806 10:53:56.960631 4826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:53:56.971598 4826 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:53:56.972997 4826 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:53:56.984086 4826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e37d0e43e5d8407ca8bfc05d45d2265a: Generated new TSK 0
I20260806 10:53:56.985165 4826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:53:57.058414 4808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:56.999339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60530 (local address 127.30.218.62:42567)
0806 10:53:56.999554 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:56.999561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:56.999886 (+ 325us) server_negotiation.cc:407] Connection header received
0806 10:53:57.000154 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:57.000158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:57.000236 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:57.000388 (+ 152us) server_negotiation.cc:226] Negotiated authn=JWT
0806 10:53:57.000892 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:57.001692 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:57.003172 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:57.003649 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:57.003781 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:57.004333 (+ 552us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 10:53:57.057607 (+ 53274us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 10:53:57.058077 (+ 470us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60530: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":58}
W20260806 10:53:57.058583 4808 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60530: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 10:53:57.058574 4834 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:53:56.999159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42567 (local address 127.0.0.1:60530)
0806 10:53:56.999774 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0806 10:53:56.999794 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:53:56.999953 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:53:57.000428 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:53:57.000433 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:53:57.000446 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0806 10:53:57.000729 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:57.000739 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:57.001851 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:57.001855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:57.003004 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:53:57.003017 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:53:57.003840 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:53:57.003848 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:53:57.003990 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:57.004031 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 10:53:57.057942 (+ 53911us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 10:53:57.058225 (+ 283us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.218.62:42567: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":413,"threads_started":1}
I20260806 10:53:57.060168 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 4739
I20260806 10:53:57.061703 4800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:53:57.161085 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 4739
2026-08-06T10:53:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (775 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (427 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (430 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:53:58 dist-test-slave-49h3 krb5kdc[4849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:53:58 dist-test-slave-49h3 krb5kdc[4849](info): set up 2 sockets
Aug 06 10:53:58 dist-test-slave-49h3 krb5kdc[4849](info): commencing operation
krb5kdc: starting...
W20260806 10:53:59.067008 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.004s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:53:59 dist-test-slave-49h3 krb5kdc[4849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:53:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:53:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:53:59.337064 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:43133
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:43133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:53:59.678411 4862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:53:59.678777 4862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:53:59.678925 4862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:53:59.686756 4862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:53:59.686914 4862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:53:59.687000 4862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:53:59.687083 4862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:53:59.687146 4862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:53:59.697896 4862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:43133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:43133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:53:59.699393 4862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:53:59.700963 4862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:53:59.713340 4868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:59.713363 4869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:53:59.713300 4871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:53:59.714534 4862 server_base.cc:1034] running on GCE node
I20260806 10:53:59.715497 4862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:53:59.717141 4862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:53:59.718369 4862 hybrid_clock.cc:648] HybridClock initialized: now 1786013639718310 us; error 51 us; skew 500 ppm
I20260806 10:53:59.718761 4862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:53:59 dist-test-slave-49h3 krb5kdc[4849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013639, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:53:59.728876 4862 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:53:59.730626 4862 webserver.cc:460] Webserver started at http://127.30.218.62:44393/ using document root <none> and password file <none>
I20260806 10:53:59.731335 4862 fs_manager.cc:359] Metadata directory not provided
I20260806 10:53:59.731446 4862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:53:59.731729 4862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:53:59.734972 4862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "9a21a90d689847b19a7f133112d5cccd"
format_stamp: "Formatted at 2026-08-06 10:53:59 on dist-test-slave-49h3"
server_key: "dc788a5423e597a071721fa29f1e5de5"
server_key_iv: "13d77915615a985f969af3dca51bf491"
server_key_version: "encryptionkey@0"
I20260806 10:53:59.735754 4862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "9a21a90d689847b19a7f133112d5cccd"
format_stamp: "Formatted at 2026-08-06 10:53:59 on dist-test-slave-49h3"
server_key: "dc788a5423e597a071721fa29f1e5de5"
server_key_iv: "13d77915615a985f969af3dca51bf491"
server_key_version: "encryptionkey@0"
I20260806 10:53:59.741331 4862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 10:53:59.745224 4878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:53:59.746767 4862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:53:59.746948 4862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "9a21a90d689847b19a7f133112d5cccd"
format_stamp: "Formatted at 2026-08-06 10:53:59 on dist-test-slave-49h3"
server_key: "dc788a5423e597a071721fa29f1e5de5"
server_key_iv: "13d77915615a985f969af3dca51bf491"
server_key_version: "encryptionkey@0"
I20260806 10:53:59.747154 4862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:53:59.769716 4862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:53:59.774170 4862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:53:59.774432 4862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:53:59.793418 4862 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:43133
I20260806 10:53:59.793438 4929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:43133 every 8 connection(s)
I20260806 10:53:59.794996 4862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:53:59.799176 4930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:53:59.804284 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 4862
I20260806 10:53:59.804556 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:53:59.805419 31592 external_mini_cluster.cc:1428] Setting key f652a07e09cfbd8a5b583588b53477cf
I20260806 10:53:59.812093 4930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a21a90d689847b19a7f133112d5cccd: Bootstrap starting.
I20260806 10:53:59.815519 4930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a21a90d689847b19a7f133112d5cccd: Neither blocks nor log segments found. Creating new log.
I20260806 10:53:59.816639 4930 log.cc:824] T 00000000000000000000000000000000 P 9a21a90d689847b19a7f133112d5cccd: Log is configured to *not* fsync() on all Append() calls
I20260806 10:53:59.819689 4930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a21a90d689847b19a7f133112d5cccd: No bootstrap required, opened a new log
Aug 06 10:53:59 dist-test-slave-49h3 krb5kdc[4849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:53:59.827814 4930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a21a90d689847b19a7f133112d5cccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a21a90d689847b19a7f133112d5cccd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 43133 } }
I20260806 10:53:59.828231 4930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a21a90d689847b19a7f133112d5cccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:53:59.828330 4930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a21a90d689847b19a7f133112d5cccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a21a90d689847b19a7f133112d5cccd, State: Initialized, Role: FOLLOWER
I20260806 10:53:59.828781 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:53:59.807868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60890 (local address 127.30.218.62:43133)
0806 10:53:59.808769 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0806 10:53:59.808782 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 10:53:59.808833 (+ 51us) server_negotiation.cc:407] Connection header received
0806 10:53:59.810831 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:53:59.810859 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:53:59.811344 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:53:59.811778 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:53:59.813305 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:59.814749 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:59.816138 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:53:59.816879 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:53:59.817055 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:53:59.821112 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:53:59.821120 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:53:59.821137 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:53:59.821181 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:53:59.824438 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:59.825169 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:59.825180 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:59.825187 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:59.825374 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:53:59.825793 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:53:59.825797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:53:59.825799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:53:59.826422 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:53:59.826726 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0806 10:53:59.827184 (+ 458us) server_negotiation.cc:299] Negotiation successful
0806 10:53:59.827630 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":403,"threads_started":1}
I20260806 10:53:59.828899 4930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a21a90d689847b19a7f133112d5cccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a21a90d689847b19a7f133112d5cccd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 43133 } }
I20260806 10:53:59.829213 4930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a21a90d689847b19a7f133112d5cccd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:53:59.829362 4930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a21a90d689847b19a7f133112d5cccd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:53:59.829576 4930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a21a90d689847b19a7f133112d5cccd [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:53:59.832242 4930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a21a90d689847b19a7f133112d5cccd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a21a90d689847b19a7f133112d5cccd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 43133 } }
I20260806 10:53:59.832675 4930 leader_election.cc:302] T 00000000000000000000000000000000 P 9a21a90d689847b19a7f133112d5cccd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a21a90d689847b19a7f133112d5cccd; no voters:
I20260806 10:53:59.833194 4930 leader_election.cc:288] T 00000000000000000000000000000000 P 9a21a90d689847b19a7f133112d5cccd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:53:59.833330 4935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a21a90d689847b19a7f133112d5cccd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:53:59.834002 4935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a21a90d689847b19a7f133112d5cccd [term 1 LEADER]: Becoming Leader. State: Replica: 9a21a90d689847b19a7f133112d5cccd, State: Running, Role: LEADER
I20260806 10:53:59.834535 4935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a21a90d689847b19a7f133112d5cccd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a21a90d689847b19a7f133112d5cccd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 43133 } }
I20260806 10:53:59.835130 4930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a21a90d689847b19a7f133112d5cccd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:53:59.837471 4936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a21a90d689847b19a7f133112d5cccd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a21a90d689847b19a7f133112d5cccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a21a90d689847b19a7f133112d5cccd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 43133 } } }
I20260806 10:53:59.837574 4937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a21a90d689847b19a7f133112d5cccd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a21a90d689847b19a7f133112d5cccd. Latest consensus state: current_term: 1 leader_uuid: "9a21a90d689847b19a7f133112d5cccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a21a90d689847b19a7f133112d5cccd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 43133 } } }
I20260806 10:53:59.837899 4936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a21a90d689847b19a7f133112d5cccd [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:59.837899 4937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a21a90d689847b19a7f133112d5cccd [sys.catalog]: This master's current role is: LEADER
I20260806 10:53:59.847926 4943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:53:59.852422 4943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:53:59.861869 4943 catalog_manager.cc:1347] Generated new cluster ID: 46af4ef60e9c487aa120235941f2ef2a
I20260806 10:53:59.862000 4943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:53:59.880113 4943 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:53:59.882265 4943 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:53:59.898483 4943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a21a90d689847b19a7f133112d5cccd: Generated new TSK 0
I20260806 10:53:59.899737 4943 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
I20260806 10:53:59.967684 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:43133
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:54:00.314034 4956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:00.314368 4956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:00.314499 4956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:00.322203 4956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:00.322357 4956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:00.322520 4956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:54:00.333304 4956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:43133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:00.334750 4956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:00.336292 4956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:00.350240 4964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:00.350401 4961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:00.350379 4962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:00.351542 4956 server_base.cc:1034] running on GCE node
I20260806 10:54:00.352296 4956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:00.353370 4956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:00.354612 4956 hybrid_clock.cc:648] HybridClock initialized: now 1786013640354550 us; error 51 us; skew 500 ppm
I20260806 10:54:00.354987 4956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:00 dist-test-slave-49h3 krb5kdc[4849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013640, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:00.358397 4956 init.cc:377] Logged in from keytab as kudu/127.30.218.1@KRBTEST.COM (short username kudu)
I20260806 10:54:00.360002 4956 webserver.cc:460] Webserver started at http://127.30.218.1:39349/ using document root <none> and password file <none>
I20260806 10:54:00.360745 4956 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:00.360858 4956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:00.361145 4956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:00.364363 4956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "5293313a366e436297eb872c02f2af6f"
format_stamp: "Formatted at 2026-08-06 10:54:00 on dist-test-slave-49h3"
server_key: "9bf182d65a664c282136584a9a0f9e87"
server_key_iv: "414dfcca1ebcd5ffd24f661074d754f9"
server_key_version: "encryptionkey@0"
I20260806 10:54:00.365223 4956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "5293313a366e436297eb872c02f2af6f"
format_stamp: "Formatted at 2026-08-06 10:54:00 on dist-test-slave-49h3"
server_key: "9bf182d65a664c282136584a9a0f9e87"
server_key_iv: "414dfcca1ebcd5ffd24f661074d754f9"
server_key_version: "encryptionkey@0"
I20260806 10:54:00.370962 4956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 10:54:00.374707 4971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:00.376191 4956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:54:00.376376 4956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "5293313a366e436297eb872c02f2af6f"
format_stamp: "Formatted at 2026-08-06 10:54:00 on dist-test-slave-49h3"
server_key: "9bf182d65a664c282136584a9a0f9e87"
server_key_iv: "414dfcca1ebcd5ffd24f661074d754f9"
server_key_version: "encryptionkey@0"
I20260806 10:54:00.376597 4956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:00.403798 4956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:00.408615 4956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:00.408912 4956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:00.409999 4956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:54:00.411513 4956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:00.411607 4956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:00.411732 4956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:00.411787 4956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:00.443260 4956 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:35039
I20260806 10:54:00.443311 5084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:35039 every 8 connection(s)
I20260806 10:54:00.444499 4956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:54:00.446765 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 4956
I20260806 10:54:00.447006 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:54:00.447716 31592 external_mini_cluster.cc:1428] Setting key b1dba8fc704c66020b1c7260b025b4ad
Aug 06 10:54:00 dist-test-slave-49h3 krb5kdc[4849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013640, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:54:00.466648 5087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:00.446984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:35839 (local address 127.30.218.62:43133)
0806 10:54:00.447476 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:00.447483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:00.448535 (+ 1052us) server_negotiation.cc:407] Connection header received
0806 10:54:00.450787 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:00.450791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:00.450856 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:00.450943 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:00.452897 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:00.453670 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:00.455684 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:00.456190 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:00.456299 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:00.460387 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:00.460393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:00.460399 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:00.460440 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:00.463473 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:00.464229 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:00.464235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:00.464238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:00.464378 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:00.464798 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:00.464803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:00.464805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:00.465122 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:00.465270 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:00.466062 (+ 792us) server_negotiation.cc:299] Negotiation successful
0806 10:54:00.466298 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":228,"threads_started":1}
I20260806 10:54:00.467514 5088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:00.447543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:43133 (local address 127.30.218.1:35839)
0806 10:54:00.448370 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:00.448420 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:00.450495 (+ 2075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:00.451183 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:00.451194 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:00.451911 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:00.452718 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:00.452743 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:00.453828 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:00.453833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:00.455547 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:00.455561 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:00.456417 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:00.456423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:00.456727 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:00.457471 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:00.457500 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:00.460127 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:00.463624 (+ 3497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:00.463631 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:00.463645 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:00.464101 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:00.464484 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:00.464487 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:00.464490 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:00.464696 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:00.465273 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:00.465279 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:00.465666 (+ 387us) client_negotiation.cc:769] Sending connection context
0806 10:54:00.465907 (+ 241us) client_negotiation.cc:240] Negotiation successful
0806 10:54:00.466343 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":319,"threads_started":1}
I20260806 10:54:00.469056 5085 heartbeater.cc:347] Connected to a master server at 127.30.218.62:43133
I20260806 10:54:00.469362 5085 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:00.470134 5085 heartbeater.cc:511] Master 127.30.218.62:43133 requested a full tablet report, sending...
I20260806 10:54:00.472302 4895 ts_manager.cc:194] Registered new tserver with Master: 5293313a366e436297eb872c02f2af6f (127.30.218.1:35039)
WARNING: no policy specified for kudu/127.30.218.2@KRBTEST.COM; defaulting to no policy
I20260806 10:54:00.473887 4895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.1@KRBTEST.COM'} at 127.30.218.1:35839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
I20260806 10:54:00.507025 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:43133
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:54:00.854947 5091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:00.855284 5091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:00.855415 5091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:00.863211 5091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:00.863345 5091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:00.863502 5091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:54:00.874434 5091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:43133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:00.875870 5091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:00.877306 5091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:00.891263 5096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:00.891783 5097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:00.891811 5099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:00.892117 5091 server_base.cc:1034] running on GCE node
I20260806 10:54:00.892802 5091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:00.893913 5091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:00.895120 5091 hybrid_clock.cc:648] HybridClock initialized: now 1786013640895074 us; error 51 us; skew 500 ppm
I20260806 10:54:00.895457 5091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:00 dist-test-slave-49h3 krb5kdc[4849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013640, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:00.898836 5091 init.cc:377] Logged in from keytab as kudu/127.30.218.2@KRBTEST.COM (short username kudu)
I20260806 10:54:00.900424 5091 webserver.cc:460] Webserver started at http://127.30.218.2:40377/ using document root <none> and password file <none>
I20260806 10:54:00.901163 5091 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:00.901294 5091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:00.901669 5091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:00.904902 5091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "c7ed9ded2938425db05d5f9cc1dafd8d"
format_stamp: "Formatted at 2026-08-06 10:54:00 on dist-test-slave-49h3"
server_key: "d38e27651b2027a70bf43b4473f8d958"
server_key_iv: "89b09aac31a661e481dc18594c56e564"
server_key_version: "encryptionkey@0"
I20260806 10:54:00.905774 5091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "c7ed9ded2938425db05d5f9cc1dafd8d"
format_stamp: "Formatted at 2026-08-06 10:54:00 on dist-test-slave-49h3"
server_key: "d38e27651b2027a70bf43b4473f8d958"
server_key_iv: "89b09aac31a661e481dc18594c56e564"
server_key_version: "encryptionkey@0"
I20260806 10:54:00.911296 5091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260806 10:54:00.915892 5106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:00.917337 5091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:54:00.917539 5091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "c7ed9ded2938425db05d5f9cc1dafd8d"
format_stamp: "Formatted at 2026-08-06 10:54:00 on dist-test-slave-49h3"
server_key: "d38e27651b2027a70bf43b4473f8d958"
server_key_iv: "89b09aac31a661e481dc18594c56e564"
server_key_version: "encryptionkey@0"
I20260806 10:54:00.917760 5091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:00.944553 5091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:00.949316 5091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:00.949669 5091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:00.950799 5091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:54:00.952358 5091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:00.952445 5091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:00.952572 5091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:00.952626 5091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:00.984668 5091 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:35005
I20260806 10:54:00.984789 5220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:35005 every 8 connection(s)
I20260806 10:54:00.985978 5091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:54:00.995141 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 5091
I20260806 10:54:00.995323 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:54:00.995743 31592 external_mini_cluster.cc:1428] Setting key f9a40d4f310a0d8d21de116e59d2f372
Aug 06 10:54:00 dist-test-slave-49h3 krb5kdc[4849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013640, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:54:01.005335 5223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:00.988451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:35635 (local address 127.30.218.62:43133)
0806 10:54:00.988862 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:00.988867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:00.989765 (+ 898us) server_negotiation.cc:407] Connection header received
0806 10:54:00.991569 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:00.991576 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:00.991657 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:00.991761 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:00.993213 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:00.993974 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:00.995504 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:00.995979 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:00.996090 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:00.999925 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:00.999930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:00.999933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:00.999964 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:01.002342 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:01.003185 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:01.003191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:01.003194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:01.003323 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:01.003771 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:01.003774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:01.003776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:01.004010 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:01.004115 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:01.004888 (+ 773us) server_negotiation.cc:299] Negotiation successful
0806 10:54:01.005066 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":218,"threads_started":1}
I20260806 10:54:01.006645 5224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:00.988895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:43133 (local address 127.30.218.2:35635)
0806 10:54:00.989607 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:00.989650 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:00.991310 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:00.991988 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:00.991996 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:00.992512 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:00.993044 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:00.993062 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:00.994121 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:00.994125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:00.995364 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:00.995386 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:00.996200 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:00.996208 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:00.996427 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:00.997134 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:00.997156 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:00.999710 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:01.002493 (+ 2783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:01.002502 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:01.002517 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:01.003085 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:01.003420 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:01.003424 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:01.003427 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:01.003680 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:01.004119 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:01.004125 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:01.004508 (+ 383us) client_negotiation.cc:769] Sending connection context
0806 10:54:01.004809 (+ 301us) client_negotiation.cc:240] Negotiation successful
0806 10:54:01.005319 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":288,"threads_started":1}
I20260806 10:54:01.007974 5221 heartbeater.cc:347] Connected to a master server at 127.30.218.62:43133
I20260806 10:54:01.008276 5221 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:01.008968 5221 heartbeater.cc:511] Master 127.30.218.62:43133 requested a full tablet report, sending...
I20260806 10:54:01.010462 4895 ts_manager.cc:194] Registered new tserver with Master: c7ed9ded2938425db05d5f9cc1dafd8d (127.30.218.2:35005)
I20260806 10:54:01.011523 4895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.2@KRBTEST.COM'} at 127.30.218.2:35635
WARNING: no policy specified for kudu/127.30.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
I20260806 10:54:01.053020 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:43133
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:54:01.401523 5227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:01.401811 5227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:01.401947 5227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:01.409186 5227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:01.409313 5227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:01.409487 5227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:54:01.419766 5227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:43133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:01.421219 5227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:01.422693 5227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:01.435917 5232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:01.435954 5233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:01.435882 5235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:01.437191 5227 server_base.cc:1034] running on GCE node
I20260806 10:54:01.437928 5227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:01.439057 5227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:01.440286 5227 hybrid_clock.cc:648] HybridClock initialized: now 1786013641440208 us; error 68 us; skew 500 ppm
I20260806 10:54:01.440645 5227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:01 dist-test-slave-49h3 krb5kdc[4849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013641, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:01.444031 5227 init.cc:377] Logged in from keytab as kudu/127.30.218.3@KRBTEST.COM (short username kudu)
I20260806 10:54:01.445749 5227 webserver.cc:460] Webserver started at http://127.30.218.3:46571/ using document root <none> and password file <none>
I20260806 10:54:01.446503 5227 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:01.446638 5227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:01.446952 5227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:01.450244 5227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "59a1699da6dc477cb425b21068341359"
format_stamp: "Formatted at 2026-08-06 10:54:01 on dist-test-slave-49h3"
server_key: "1e99c552c6b3ae26fd1508273ccdb974"
server_key_iv: "a690a7345d0c0782d04c51c3d31714aa"
server_key_version: "encryptionkey@0"
I20260806 10:54:01.451188 5227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "59a1699da6dc477cb425b21068341359"
format_stamp: "Formatted at 2026-08-06 10:54:01 on dist-test-slave-49h3"
server_key: "1e99c552c6b3ae26fd1508273ccdb974"
server_key_iv: "a690a7345d0c0782d04c51c3d31714aa"
server_key_version: "encryptionkey@0"
I20260806 10:54:01.456950 5227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 10:54:01.460906 5242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:01.462396 5227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:54:01.462571 5227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "59a1699da6dc477cb425b21068341359"
format_stamp: "Formatted at 2026-08-06 10:54:01 on dist-test-slave-49h3"
server_key: "1e99c552c6b3ae26fd1508273ccdb974"
server_key_iv: "a690a7345d0c0782d04c51c3d31714aa"
server_key_version: "encryptionkey@0"
I20260806 10:54:01.462807 5227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:01.477267 5085 heartbeater.cc:503] Master 127.30.218.62:43133 was elected leader, sending a full tablet report...
I20260806 10:54:01.485543 5227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:01.490025 5227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:01.490355 5227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:01.491389 5227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:54:01.492887 5227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:01.492976 5227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:01.493113 5227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:01.493170 5227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:01.524957 5227 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:46813
I20260806 10:54:01.525002 5355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:46813 every 8 connection(s)
I20260806 10:54:01.526542 5227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:54:01.530320 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 5227
I20260806 10:54:01.530514 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:54:01.530969 31592 external_mini_cluster.cc:1428] Setting key 34b3ef78ec99840cd73f220d16e7935e
Aug 06 10:54:01 dist-test-slave-49h3 krb5kdc[4849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013641, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:54:01.546901 5358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:01.529322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:40499 (local address 127.30.218.62:43133)
0806 10:54:01.529826 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:01.529832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:01.530891 (+ 1059us) server_negotiation.cc:407] Connection header received
0806 10:54:01.533103 (+ 2212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:01.533107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:01.533177 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:01.533266 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:01.535050 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:01.535911 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:01.537476 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:01.537872 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:01.537969 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:01.541188 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:01.541193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:01.541196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:01.541231 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:01.543824 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:01.544487 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:01.544492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:01.544495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:01.544638 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:01.545071 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:01.545076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:01.545080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:01.545417 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:01.545602 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:01.546415 (+ 813us) server_negotiation.cc:299] Negotiation successful
0806 10:54:01.546619 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":262,"threads_started":1}
I20260806 10:54:01.547873 5359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:01.529876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:43133 (local address 127.30.218.3:40499)
0806 10:54:01.530719 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:01.530764 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:01.532831 (+ 2067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:01.533565 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:01.533576 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:01.534299 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:01.534874 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:01.534895 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:01.536042 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:01.536046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:01.537339 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:01.537351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:01.538045 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:01.538049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:01.538258 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:01.538867 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:01.538890 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:01.540986 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:01.543944 (+ 2958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:01.543950 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:01.543964 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:01.544388 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:01.544750 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:01.544754 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:01.544757 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:01.544975 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:01.545565 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:01.545571 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:01.545955 (+ 384us) client_negotiation.cc:769] Sending connection context
0806 10:54:01.546302 (+ 347us) client_negotiation.cc:240] Negotiation successful
0806 10:54:01.546822 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":328,"threads_started":1}
I20260806 10:54:01.549067 5356 heartbeater.cc:347] Connected to a master server at 127.30.218.62:43133
I20260806 10:54:01.549357 5356 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:01.550035 5356 heartbeater.cc:511] Master 127.30.218.62:43133 requested a full tablet report, sending...
I20260806 10:54:01.551380 4895 ts_manager.cc:194] Registered new tserver with Master: 59a1699da6dc477cb425b21068341359 (127.30.218.3:46813)
I20260806 10:54:01.552201 4895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.3@KRBTEST.COM'} at 127.30.218.3:40499
I20260806 10:54:01.558583 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/test-admin.keytab.
Aug 06 10:54:01 dist-test-slave-49h3 krb5kdc[4849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:01.591364 31592 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 10:54:01.597082 5364 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:01.594380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56886 (local address 127.30.218.1:35039)
0806 10:54:01.595065 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:01.595072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:01.595099 (+ 27us) server_negotiation.cc:407] Connection header received
0806 10:54:01.595159 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:01.595166 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:01.595376 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:01.595553 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:01.596650 (+ 1097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56886: BlockingRecv error: recv got EOF from 127.0.0.1:56886 (error 108)
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":334,"threads_started":1}
W20260806 10:54:01.597263 31592 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/krb5cc' of type FILE
Aug 06 10:54:01 dist-test-slave-49h3 krb5kdc[4849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:01.598938 31592 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 10:54:01 dist-test-slave-49h3 krb5kdc[4849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:01.600373 31592 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 10:54:01 dist-test-slave-49h3 krb5kdc[4849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.1@KRBTEST.COM
I20260806 10:54:01.617658 5364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:01.602459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56894 (local address 127.30.218.1:35039)
0806 10:54:01.602687 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:01.602694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:01.602814 (+ 120us) server_negotiation.cc:407] Connection header received
0806 10:54:01.603004 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:01.603009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:01.603080 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:01.603169 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:01.604441 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:01.605483 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:01.606814 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:01.607429 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:01.607568 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:01.611540 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:01.611555 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:01.611572 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:01.611628 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:01.614484 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:01.615327 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:01.615335 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:01.615338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:01.615490 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:01.615889 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:01.615892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:01.615895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:01.616311 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:01.616616 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:01.616978 (+ 362us) server_negotiation.cc:299] Negotiation successful
0806 10:54:01.617280 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260806 10:54:01.623879 5364 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:01.621937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56898 (local address 127.30.218.1:35039)
0806 10:54:01.622169 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:01.622175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:01.622479 (+ 304us) server_negotiation.cc:407] Connection header received
0806 10:54:01.622642 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:01.622647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:01.622713 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:01.622796 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:01.623621 (+ 825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56898: BlockingRecv error: recv got EOF from 127.0.0.1:56898 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260806 10:54:01.624456 31592 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/krb5cc)
Aug 06 10:54:01 dist-test-slave-49h3 krb5kdc[4849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:01.626014 31592 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 10:54:01 dist-test-slave-49h3 krb5kdc[4849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:01.627264 31592 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 10:54:01 dist-test-slave-49h3 krb5kdc[4849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.1@KRBTEST.COM
I20260806 10:54:01.641671 5364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:01.628624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56910 (local address 127.30.218.1:35039)
0806 10:54:01.628855 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:01.628861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:01.629059 (+ 198us) server_negotiation.cc:407] Connection header received
0806 10:54:01.629193 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:01.629197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:01.629269 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:01.629353 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:01.630389 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:01.631159 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:01.632393 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:01.632846 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:01.632936 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:01.636570 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:01.636575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:01.636579 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:01.636617 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:01.639328 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:01.639999 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:01.640003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:01.640005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:01.640132 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:01.640517 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:01.640520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:01.640522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:01.640795 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:01.640988 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:01.641224 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 10:54:01.641394 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260806 10:54:01.646045 5364 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:01.644410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56920 (local address 127.30.218.1:35039)
0806 10:54:01.644609 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:01.644614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:01.644816 (+ 202us) server_negotiation.cc:407] Connection header received
0806 10:54:01.644942 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:01.644945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:01.645016 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:01.645091 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:01.645798 (+ 707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56920: BlockingRecv error: recv got EOF from 127.0.0.1:56920 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260806 10:54:01.646205 31592 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786013589554914-31592-0/krb5kdc/krb5cc)
Aug 06 10:54:01 dist-test-slave-49h3 krb5kdc[4849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:01.647715 31592 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 10:54:01 dist-test-slave-49h3 krb5kdc[4849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:01.648959 31592 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 10:54:01 dist-test-slave-49h3 krb5kdc[4849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.1@KRBTEST.COM
I20260806 10:54:01.663795 5364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:01.650326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56924 (local address 127.30.218.1:35039)
0806 10:54:01.650550 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:01.650556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:01.650786 (+ 230us) server_negotiation.cc:407] Connection header received
0806 10:54:01.650929 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:01.650933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:01.650995 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:01.651064 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:01.652019 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:01.652871 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:01.654242 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:01.654713 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:01.654815 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:01.658476 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:01.658482 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:01.658485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:01.658523 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:01.661084 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:01.661809 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:01.661814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:01.661816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:01.661974 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:01.662322 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:01.662325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:01.662327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:01.662624 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:01.662872 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:01.663252 (+ 380us) server_negotiation.cc:299] Negotiation successful
0806 10:54:01.663475 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 10:54:01.665407 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 4956
I20260806 10:54:01.676802 5364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:01.666150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.30.218.1:35039)
0806 10:54:01.666317 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:01.666321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:01.666625 (+ 304us) server_negotiation.cc:407] Connection header received
0806 10:54:01.666744 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:01.666747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:01.666826 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:01.666898 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:01.667871 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:01.668567 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:01.669761 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:01.670171 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:01.670260 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:01.672132 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:01.672137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:01.672141 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:01.672179 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:01.674564 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:01.675200 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:01.675203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:01.675206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:01.675330 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:01.675606 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:01.675608 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:01.675610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:01.675858 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:01.676048 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:01.676374 (+ 326us) server_negotiation.cc:299] Negotiation successful
0806 10:54:01.676536 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 10:54:01.677402 5080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:01.761729 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 4956
I20260806 10:54:01.778509 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 5091
Aug 06 10:54:01 dist-test-slave-49h3 krb5kdc[4849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.2@KRBTEST.COM
I20260806 10:54:01.794651 5377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:01.779395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45890 (local address 127.30.218.2:35005)
0806 10:54:01.779913 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:01.779920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:01.779944 (+ 24us) server_negotiation.cc:407] Connection header received
0806 10:54:01.780008 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:01.780017 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:01.780295 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:01.780437 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:01.781392 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:01.782481 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:01.783737 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:01.784337 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:01.784446 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:01.788454 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:01.788473 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:01.788492 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:01.788546 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:01.791618 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:01.792404 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:01.792411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:01.792413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:01.792569 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:01.792978 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:01.792981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:01.792983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:01.793409 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:01.793710 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:01.794063 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 10:54:01.794280 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":260,"threads_started":1}
I20260806 10:54:01.796118 5215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:01.881402 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 5091
I20260806 10:54:01.898576 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 5227
Aug 06 10:54:01 dist-test-slave-49h3 krb5kdc[4849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.3@KRBTEST.COM
I20260806 10:54:01.916491 5379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:01.899733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34926 (local address 127.30.218.3:46813)
0806 10:54:01.900483 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:01.900491 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:01.900527 (+ 36us) server_negotiation.cc:407] Connection header received
0806 10:54:01.900595 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:01.900602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:01.900849 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:01.900996 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:01.902637 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:01.903739 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:01.905117 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:01.905806 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:01.905923 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:01.910179 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:01.910196 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:01.910213 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:01.910259 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:01.913015 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:01.914078 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:01.914089 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:01.914093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:01.914251 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:01.914636 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:01.914640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:01.914642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:01.915112 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:01.915449 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:01.915759 (+ 310us) server_negotiation.cc:299] Negotiation successful
0806 10:54:01.916075 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":382,"threads_started":1}
I20260806 10:54:01.918085 5351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:02.003531 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 5227
I20260806 10:54:02.020130 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 4862
I20260806 10:54:02.021397 4925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:02.129416 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 4862
2026-08-06T10:54:02Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4118 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:54:02 dist-test-slave-49h3 krb5kdc[5384](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:54:02 dist-test-slave-49h3 krb5kdc[5384](info): set up 2 sockets
Aug 06 10:54:02 dist-test-slave-49h3 krb5kdc[5384](info): commencing operation
krb5kdc: starting...
W20260806 10:54:03.195966 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:54:03 dist-test-slave-49h3 krb5kdc[5384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:54:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:54:03Z Disabled control of system clock
WARNING: no policy specified for oryx/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.62.keytab.
Entry for principal oryx/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.62.keytab.
I20260806 10:54:03.477885 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:34161
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.62.keytab
--principal=oryx/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:34161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:54:03.821303 5397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:03.821741 5397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:03.821889 5397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:03.829804 5397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:54:03.829954 5397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:03.830039 5397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:03.830127 5397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:54:03.830194 5397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:54:03.840950 5397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:34161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.62.keytab
--principal=oryx/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:34161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:03.842533 5397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:03.844075 5397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:03.856715 5402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:03.856683 5403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:03.856925 5405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:03.857801 5397 server_base.cc:1034] running on GCE node
I20260806 10:54:03.858703 5397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:03.860343 5397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:03.861604 5397 hybrid_clock.cc:648] HybridClock initialized: now 1786013643861541 us; error 50 us; skew 500 ppm
I20260806 10:54:03.862028 5397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:03 dist-test-slave-49h3 krb5kdc[5384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013643, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:03.865608 5397 init.cc:377] Logged in from keytab as oryx/127.30.218.62@KRBTEST.COM (short username oryx)
I20260806 10:54:03.867408 5397 webserver.cc:460] Webserver started at http://127.30.218.62:43121/ using document root <none> and password file <none>
I20260806 10:54:03.868193 5397 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:03.868336 5397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:03.868669 5397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:03.872195 5397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "4b824ac0a7c94632b4d0aea8540f15bd"
format_stamp: "Formatted at 2026-08-06 10:54:03 on dist-test-slave-49h3"
server_key: "32cb13e68393d63d47669b0145521a83"
server_key_iv: "5c642a90288e6871a6a6142726f29a80"
server_key_version: "encryptionkey@0"
I20260806 10:54:03.873104 5397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "4b824ac0a7c94632b4d0aea8540f15bd"
format_stamp: "Formatted at 2026-08-06 10:54:03 on dist-test-slave-49h3"
server_key: "32cb13e68393d63d47669b0145521a83"
server_key_iv: "5c642a90288e6871a6a6142726f29a80"
server_key_version: "encryptionkey@0"
I20260806 10:54:03.879261 5397 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260806 10:54:03.883289 5412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:03.884842 5397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 10:54:03.885037 5397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "4b824ac0a7c94632b4d0aea8540f15bd"
format_stamp: "Formatted at 2026-08-06 10:54:03 on dist-test-slave-49h3"
server_key: "32cb13e68393d63d47669b0145521a83"
server_key_iv: "5c642a90288e6871a6a6142726f29a80"
server_key_version: "encryptionkey@0"
I20260806 10:54:03.885259 5397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:03.901194 5397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:03.905900 5397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:03.906199 5397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:03.926198 5397 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:34161
I20260806 10:54:03.926193 5464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:34161 every 8 connection(s)
I20260806 10:54:03.927781 5397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:54:03.931747 5465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:03.936125 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 5397
I20260806 10:54:03.936393 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:54:03.937273 31592 external_mini_cluster.cc:1428] Setting key 18e139cca9b9fc176d4cb12b6f7830a9
I20260806 10:54:03.946141 5465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd: Bootstrap starting.
I20260806 10:54:03.950387 5465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:03.952001 5465 log.cc:824] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd: Log is configured to *not* fsync() on all Append() calls
I20260806 10:54:03.955685 5465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd: No bootstrap required, opened a new log
Aug 06 10:54:03 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM
I20260806 10:54:03.965828 5465 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } }
I20260806 10:54:03.966439 5465 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:03.966557 5465 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b824ac0a7c94632b4d0aea8540f15bd, State: Initialized, Role: FOLLOWER
I20260806 10:54:03.967358 5465 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } }
I20260806 10:54:03.967705 5465 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:54:03.967873 5465 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:54:03.968057 5465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:03.970567 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:03.939541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58242 (local address 127.30.218.62:34161)
0806 10:54:03.944996 (+ 5455us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:03.945019 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:03.945707 (+ 688us) server_negotiation.cc:407] Connection header received
0806 10:54:03.948248 (+ 2541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:03.948282 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:03.948870 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:03.949351 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:03.950889 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:03.952935 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:03.954468 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:03.955525 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:03.955737 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:03.960196 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:03.960209 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:03.960235 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:03.960312 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:03.964786 (+ 4474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:03.965898 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:03.965910 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:03.965920 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:03.966180 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:03.966732 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:03.966737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:03.966740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:03.967568 (+ 828us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:03.967930 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:03.968418 (+ 488us) server_negotiation.cc:299] Negotiation successful
0806 10:54:03.968961 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5143,"thread_start_us":4879,"threads_started":1}
I20260806 10:54:03.971534 5465 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } }
I20260806 10:54:03.972146 5465 leader_election.cc:302] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b824ac0a7c94632b4d0aea8540f15bd; no voters:
I20260806 10:54:03.972818 5465 leader_election.cc:288] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:54:03.972994 5470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:54:03.974330 5470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [term 1 LEADER]: Becoming Leader. State: Replica: 4b824ac0a7c94632b4d0aea8540f15bd, State: Running, Role: LEADER
I20260806 10:54:03.974910 5470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } }
I20260806 10:54:03.975337 5465 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:54:03.977967 5472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b824ac0a7c94632b4d0aea8540f15bd. Latest consensus state: current_term: 1 leader_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } }
I20260806 10:54:03.978404 5472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [sys.catalog]: This master's current role is: LEADER
I20260806 10:54:03.979352 5471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } }
I20260806 10:54:03.979725 5471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [sys.catalog]: This master's current role is: LEADER
I20260806 10:54:03.980798 5476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:54:03.986267 5476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:54:03.997154 5476 catalog_manager.cc:1347] Generated new cluster ID: fd65f6dd547845159ea0ec495152a496
I20260806 10:54:03.997303 5476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:54:04.012930 5476 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:54:04.014487 5476 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:54:04.028807 5476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd: Generated new TSK 0
I20260806 10:54:04.029879 5476 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.30.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.1.keytab.
Entry for principal oryx/127.30.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.1.keytab.
I20260806 10:54:04.115682 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:34161
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.1.keytab
--principal=oryx/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:54:04.486855 5491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:04.487174 5491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:04.487315 5491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:04.494725 5491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:04.494849 5491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:04.494977 5491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:54:04.505237 5491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.1.keytab
--principal=oryx/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:34161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:04.506661 5491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:04.508075 5491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:04.522083 5496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:04.522015 5499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:04.522042 5497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:04.522763 5491 server_base.cc:1034] running on GCE node
I20260806 10:54:04.523444 5491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:04.524470 5491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:04.525667 5491 hybrid_clock.cc:648] HybridClock initialized: now 1786013644525610 us; error 50 us; skew 500 ppm
I20260806 10:54:04.526008 5491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:04 dist-test-slave-49h3 krb5kdc[5384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013644, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:04.529693 5491 init.cc:377] Logged in from keytab as oryx/127.30.218.1@KRBTEST.COM (short username oryx)
I20260806 10:54:04.531584 5491 webserver.cc:460] Webserver started at http://127.30.218.1:38781/ using document root <none> and password file <none>
I20260806 10:54:04.532366 5491 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:04.532497 5491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:04.532807 5491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:04.536101 5491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "3478ba34809347f3950b6b5326dd157d"
format_stamp: "Formatted at 2026-08-06 10:54:04 on dist-test-slave-49h3"
server_key: "e30a56dd22389286f85d08211a97e265"
server_key_iv: "6388040811b2f1b18004d72e96a8699b"
server_key_version: "encryptionkey@0"
I20260806 10:54:04.537087 5491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "3478ba34809347f3950b6b5326dd157d"
format_stamp: "Formatted at 2026-08-06 10:54:04 on dist-test-slave-49h3"
server_key: "e30a56dd22389286f85d08211a97e265"
server_key_iv: "6388040811b2f1b18004d72e96a8699b"
server_key_version: "encryptionkey@0"
I20260806 10:54:04.543108 5491 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260806 10:54:04.547145 5506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:04.548707 5491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:54:04.548897 5491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "3478ba34809347f3950b6b5326dd157d"
format_stamp: "Formatted at 2026-08-06 10:54:04 on dist-test-slave-49h3"
server_key: "e30a56dd22389286f85d08211a97e265"
server_key_iv: "6388040811b2f1b18004d72e96a8699b"
server_key_version: "encryptionkey@0"
I20260806 10:54:04.549139 5491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:04.583570 5491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:04.588346 5491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:04.588699 5491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:04.589920 5491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:54:04.591434 5491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:04.591521 5491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:04.591658 5491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:04.591712 5491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:04.625483 5491 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:34939
I20260806 10:54:04.625560 5619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:34939 every 8 connection(s)
I20260806 10:54:04.626894 5491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:54:04.630496 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 5491
I20260806 10:54:04.630779 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:54:04.631625 31592 external_mini_cluster.cc:1428] Setting key c9207cf70812b8acd277220b30bdc84f
Aug 06 10:54:04 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013644, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM
I20260806 10:54:04.649991 5622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:04.629743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:35725 (local address 127.30.218.62:34161)
0806 10:54:04.630245 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:04.630250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:04.631517 (+ 1267us) server_negotiation.cc:407] Connection header received
0806 10:54:04.633905 (+ 2388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:04.633911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:04.633997 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:04.634091 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:04.635819 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:04.636568 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:04.638305 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:04.638722 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:04.638842 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:04.643098 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:04.643104 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:04.643107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:04.643149 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:04.646348 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:04.647297 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:04.647303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:04.647305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:04.647452 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:04.647951 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:04.647955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:04.647957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:04.648272 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:04.648421 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:04.649279 (+ 858us) server_negotiation.cc:299] Negotiation successful
0806 10:54:04.649605 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":268,"threads_started":1}
I20260806 10:54:04.651103 5623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:04.630384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.30.218.1:35725)
0806 10:54:04.631251 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:04.631297 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:04.633556 (+ 2259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:04.634385 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:04.634394 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:04.634981 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:04.635621 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:04.635641 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:04.636753 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:04.636757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:04.638135 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:04.638150 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:04.638983 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:04.638991 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:04.639249 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:04.639957 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:04.639989 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:04.642784 (+ 2795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:04.646565 (+ 3781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:04.646574 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:04.646591 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:04.647134 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:04.647578 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:04.647583 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:04.647587 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:04.647818 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:04.648422 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:04.648432 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:04.648839 (+ 407us) client_negotiation.cc:769] Sending connection context
0806 10:54:04.649143 (+ 304us) client_negotiation.cc:240] Negotiation successful
0806 10:54:04.649810 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":343,"threads_started":1}
I20260806 10:54:04.653002 5620 heartbeater.cc:347] Connected to a master server at 127.30.218.62:34161
I20260806 10:54:04.653347 5620 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:04.654511 5620 heartbeater.cc:511] Master 127.30.218.62:34161 requested a full tablet report, sending...
I20260806 10:54:04.656880 5430 ts_manager.cc:194] Registered new tserver with Master: 3478ba34809347f3950b6b5326dd157d (127.30.218.1:34939)
I20260806 10:54:04.658500 5430 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.218.1@KRBTEST.COM'} at 127.30.218.1:35725
WARNING: no policy specified for oryx/127.30.218.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.2.keytab.
Entry for principal oryx/127.30.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.2.keytab.
I20260806 10:54:04.693161 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:34161
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.2.keytab
--principal=oryx/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:54:05.043787 5626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:05.044124 5626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:05.044258 5626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:05.052069 5626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:05.052213 5626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:05.052363 5626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:54:05.062937 5626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.2.keytab
--principal=oryx/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:34161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:05.064371 5626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:05.065894 5626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:05.079456 5632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:05.079517 5634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:05.079483 5631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:05.080442 5626 server_base.cc:1034] running on GCE node
I20260806 10:54:05.081175 5626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:05.082372 5626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:05.083606 5626 hybrid_clock.cc:648] HybridClock initialized: now 1786013645083555 us; error 49 us; skew 500 ppm
I20260806 10:54:05.083940 5626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:05 dist-test-slave-49h3 krb5kdc[5384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013645, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:05.091832 5626 init.cc:377] Logged in from keytab as oryx/127.30.218.2@KRBTEST.COM (short username oryx)
I20260806 10:54:05.093565 5626 webserver.cc:460] Webserver started at http://127.30.218.2:36785/ using document root <none> and password file <none>
I20260806 10:54:05.094339 5626 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:05.094475 5626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:05.094805 5626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:05.098219 5626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "ef6b396ce9db43c49069415d3b6881df"
format_stamp: "Formatted at 2026-08-06 10:54:05 on dist-test-slave-49h3"
server_key: "4e09613aee4fc4024e5d2ea53182f2e6"
server_key_iv: "7af5f3c6841b397a74a91d42939f3239"
server_key_version: "encryptionkey@0"
I20260806 10:54:05.099141 5626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "ef6b396ce9db43c49069415d3b6881df"
format_stamp: "Formatted at 2026-08-06 10:54:05 on dist-test-slave-49h3"
server_key: "4e09613aee4fc4024e5d2ea53182f2e6"
server_key_iv: "7af5f3c6841b397a74a91d42939f3239"
server_key_version: "encryptionkey@0"
I20260806 10:54:05.105082 5626 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 10:54:05.109144 5641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:05.110873 5626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:54:05.111079 5626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "ef6b396ce9db43c49069415d3b6881df"
format_stamp: "Formatted at 2026-08-06 10:54:05 on dist-test-slave-49h3"
server_key: "4e09613aee4fc4024e5d2ea53182f2e6"
server_key_iv: "7af5f3c6841b397a74a91d42939f3239"
server_key_version: "encryptionkey@0"
I20260806 10:54:05.111420 5626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:05.132040 5626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:05.136941 5626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:05.137252 5626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:05.138397 5626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:54:05.139920 5626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:05.140005 5626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:05.140107 5626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:05.140151 5626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:05.172501 5626 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:42745
I20260806 10:54:05.172613 5753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:42745 every 8 connection(s)
I20260806 10:54:05.173995 5626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:54:05.181665 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 5626
I20260806 10:54:05.181886 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:54:05.182360 31592 external_mini_cluster.cc:1428] Setting key 64234b10c465ee286477048f1ba8d8cc
Aug 06 10:54:05 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013645, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM
I20260806 10:54:05.196380 5757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:05.177054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:38851 (local address 127.30.218.62:34161)
0806 10:54:05.177560 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:05.177565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:05.178776 (+ 1211us) server_negotiation.cc:407] Connection header received
0806 10:54:05.180742 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:05.180748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:05.180838 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:05.180940 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:05.182940 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:05.183819 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:05.185659 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:05.186129 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:05.186248 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:05.190176 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:05.190182 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:05.190187 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:05.190227 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:05.193007 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:05.193912 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:05.193916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:05.193919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:05.194058 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:05.194513 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:05.194517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:05.194519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:05.194829 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:05.194969 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:05.195740 (+ 771us) server_negotiation.cc:299] Negotiation successful
0806 10:54:05.195993 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":221,"threads_started":1}
I20260806 10:54:05.197387 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:05.177753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.30.218.2:38851)
0806 10:54:05.178591 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:05.178631 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:05.180420 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:05.181251 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:05.181262 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:05.181905 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:05.182739 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:05.182764 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:05.183976 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:05.183980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:05.185508 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:05.185523 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:05.186378 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:05.186385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:05.186627 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:05.187309 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:05.187339 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:05.189876 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:05.193201 (+ 3325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:05.193212 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:05.193227 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:05.193775 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:05.194174 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:05.194187 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:05.194190 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:05.194393 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:05.194958 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:05.194966 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:05.195348 (+ 382us) client_negotiation.cc:769] Sending connection context
0806 10:54:05.195602 (+ 254us) client_negotiation.cc:240] Negotiation successful
0806 10:54:05.196212 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":322,"threads_started":1}
I20260806 10:54:05.198861 5754 heartbeater.cc:347] Connected to a master server at 127.30.218.62:34161
I20260806 10:54:05.199167 5754 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:05.199966 5754 heartbeater.cc:511] Master 127.30.218.62:34161 requested a full tablet report, sending...
I20260806 10:54:05.201596 5430 ts_manager.cc:194] Registered new tserver with Master: ef6b396ce9db43c49069415d3b6881df (127.30.218.2:42745)
I20260806 10:54:05.202715 5430 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.218.2@KRBTEST.COM'} at 127.30.218.2:38851
WARNING: no policy specified for oryx/127.30.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.3.keytab.
Entry for principal oryx/127.30.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.3.keytab.
I20260806 10:54:05.243566 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:34161
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.3.keytab
--principal=oryx/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:54:05.596009 5761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:05.596345 5761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:05.596448 5761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:05.604135 5761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:05.604285 5761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:05.604406 5761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:54:05.614986 5761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.3.keytab
--principal=oryx/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:34161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:05.616456 5761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:05.617955 5761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:05.632473 5766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:05.632572 5769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:05.632576 5767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:05.633297 5761 server_base.cc:1034] running on GCE node
I20260806 10:54:05.633978 5761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:05.635032 5761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:05.636256 5761 hybrid_clock.cc:648] HybridClock initialized: now 1786013645636193 us; error 57 us; skew 500 ppm
I20260806 10:54:05.636592 5761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:05 dist-test-slave-49h3 krb5kdc[5384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013645, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:05.640025 5761 init.cc:377] Logged in from keytab as oryx/127.30.218.3@KRBTEST.COM (short username oryx)
I20260806 10:54:05.641993 5761 webserver.cc:460] Webserver started at http://127.30.218.3:39709/ using document root <none> and password file <none>
I20260806 10:54:05.642819 5761 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:05.642949 5761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:05.643291 5761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:05.646785 5761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "b9d816dd0f9147479f65e57e4a4f33ba"
format_stamp: "Formatted at 2026-08-06 10:54:05 on dist-test-slave-49h3"
server_key: "c28b6f9cb18fd50a26111e6748a905c5"
server_key_iv: "e68a00c14b21b7fd7cf5fcc6e7ed34d8"
server_key_version: "encryptionkey@0"
I20260806 10:54:05.647692 5761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "b9d816dd0f9147479f65e57e4a4f33ba"
format_stamp: "Formatted at 2026-08-06 10:54:05 on dist-test-slave-49h3"
server_key: "c28b6f9cb18fd50a26111e6748a905c5"
server_key_iv: "e68a00c14b21b7fd7cf5fcc6e7ed34d8"
server_key_version: "encryptionkey@0"
I20260806 10:54:05.653863 5761 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 10:54:05.657775 5776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:05.659221 5761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:54:05.659382 5761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "b9d816dd0f9147479f65e57e4a4f33ba"
format_stamp: "Formatted at 2026-08-06 10:54:05 on dist-test-slave-49h3"
server_key: "c28b6f9cb18fd50a26111e6748a905c5"
server_key_iv: "e68a00c14b21b7fd7cf5fcc6e7ed34d8"
server_key_version: "encryptionkey@0"
I20260806 10:54:05.659582 5761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:05.661906 5620 heartbeater.cc:503] Master 127.30.218.62:34161 was elected leader, sending a full tablet report...
I20260806 10:54:05.683465 5761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:05.688010 5761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:05.688297 5761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:05.689280 5761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:54:05.690709 5761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:05.690795 5761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:05.690917 5761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:05.690970 5761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:05.721894 5761 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:36483
I20260806 10:54:05.721946 5890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:36483 every 8 connection(s)
I20260806 10:54:05.723234 5761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:54:05.732005 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 5761
I20260806 10:54:05.732200 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:54:05.732654 31592 external_mini_cluster.cc:1428] Setting key e8a145b69ba5ff200c3b344d62832fef
Aug 06 10:54:05 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013645, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM
I20260806 10:54:05.743585 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:05.725813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:55157 (local address 127.30.218.62:34161)
0806 10:54:05.726273 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:05.726277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:05.727125 (+ 848us) server_negotiation.cc:407] Connection header received
0806 10:54:05.728974 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:05.728980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:05.729073 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:05.729175 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:05.730763 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:05.731529 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:05.733201 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:05.733675 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:05.733806 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:05.737700 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:05.737705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:05.737709 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:05.737747 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:05.740348 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:05.741147 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:05.741151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:05.741153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:05.741283 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:05.741742 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:05.741746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:05.741749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:05.742061 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:05.742197 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:05.743076 (+ 879us) server_negotiation.cc:299] Negotiation successful
0806 10:54:05.743296 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":220,"threads_started":1}
I20260806 10:54:05.744572 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:05.726261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.30.218.3:55157)
0806 10:54:05.726964 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:05.727008 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:05.728698 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:05.729421 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:05.729430 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:05.730012 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:05.730587 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:05.730604 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:05.731702 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:05.731706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:05.733048 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:05.733062 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:05.733902 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:05.733910 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:05.734243 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:05.734968 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:05.734993 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:05.737427 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:05.740521 (+ 3094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:05.740529 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:05.740545 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:05.741035 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:05.741373 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:05.741377 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:05.741379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:05.741634 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:05.742235 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:05.742242 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:05.742625 (+ 383us) client_negotiation.cc:769] Sending connection context
0806 10:54:05.742896 (+ 271us) client_negotiation.cc:240] Negotiation successful
0806 10:54:05.743372 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":274,"threads_started":1}
I20260806 10:54:05.746073 5891 heartbeater.cc:347] Connected to a master server at 127.30.218.62:34161
I20260806 10:54:05.746402 5891 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:05.747319 5891 heartbeater.cc:511] Master 127.30.218.62:34161 requested a full tablet report, sending...
I20260806 10:54:05.748909 5430 ts_manager.cc:194] Registered new tserver with Master: b9d816dd0f9147479f65e57e4a4f33ba (127.30.218.3:36483)
I20260806 10:54:05.749809 5430 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.218.3@KRBTEST.COM'} at 127.30.218.3:55157
I20260806 10:54:05.761263 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.30.218.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.218.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.218.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.61.keytab.
Entry for principal oryx/127.30.218.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.61.keytab.
I20260806 10:54:05.821089 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.61:45205
--webserver_interface=127.30.218.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.61.keytab
--principal=oryx/127.30.218.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:34161,127.30.218.61:45205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:54:06.168804 5897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:06.169126 5897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:06.169267 5897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:06.176811 5897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:54:06.176939 5897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:06.177016 5897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:06.177091 5897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:54:06.177151 5897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:54:06.187924 5897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:34161,127.30.218.61:45205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.61.keytab
--principal=oryx/127.30.218.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.61:45205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.218.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
W20260806 10:54:06.188369 5897 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.30.218.62:34161,127.30.218.61:45205'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 10:54:06.189602 5897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:06.191061 5897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:06.203336 5903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:06.203336 5902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:06.203365 5905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:06.204209 5897 server_base.cc:1034] running on GCE node
I20260806 10:54:06.205129 5897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:06.205554 5754 heartbeater.cc:503] Master 127.30.218.62:34161 was elected leader, sending a full tablet report...
I20260806 10:54:06.206976 5897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:06.208211 5897 hybrid_clock.cc:648] HybridClock initialized: now 1786013646208126 us; error 67 us; skew 500 ppm
I20260806 10:54:06.208575 5897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:06 dist-test-slave-49h3 krb5kdc[5384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013646, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:06.211968 5897 init.cc:377] Logged in from keytab as oryx/127.30.218.61@KRBTEST.COM (short username oryx)
I20260806 10:54:06.213642 5897 webserver.cc:460] Webserver started at http://127.30.218.61:34239/ using document root <none> and password file <none>
I20260806 10:54:06.214461 5897 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:06.214603 5897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:06.214928 5897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:06.218411 5897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/data/instance:
uuid: "f5fa1e7cf26345d2b7538c6d34ea8149"
format_stamp: "Formatted at 2026-08-06 10:54:06 on dist-test-slave-49h3"
server_key: "7cda9c12a6f455d8b24e08f5a6ec6fc2"
server_key_iv: "1c6f6497be0f93739e433fdf4ab32aa1"
server_key_version: "encryptionkey@0"
I20260806 10:54:06.219305 5897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/wal/instance:
uuid: "f5fa1e7cf26345d2b7538c6d34ea8149"
format_stamp: "Formatted at 2026-08-06 10:54:06 on dist-test-slave-49h3"
server_key: "7cda9c12a6f455d8b24e08f5a6ec6fc2"
server_key_iv: "1c6f6497be0f93739e433fdf4ab32aa1"
server_key_version: "encryptionkey@0"
I20260806 10:54:06.225168 5897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260806 10:54:06.229177 5912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:06.230736 5897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:54:06.230932 5897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/wal
uuid: "f5fa1e7cf26345d2b7538c6d34ea8149"
format_stamp: "Formatted at 2026-08-06 10:54:06 on dist-test-slave-49h3"
server_key: "7cda9c12a6f455d8b24e08f5a6ec6fc2"
server_key_iv: "1c6f6497be0f93739e433fdf4ab32aa1"
server_key_version: "encryptionkey@0"
I20260806 10:54:06.231170 5897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:06.251385 5897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:06.255815 5897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:06.256080 5897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:06.274360 5897 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.61:45205
I20260806 10:54:06.274371 5964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.61:45205 every 8 connection(s)
I20260806 10:54:06.275839 5897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/data/info.pb
I20260806 10:54:06.279403 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 5897
I20260806 10:54:06.279822 5965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:06.290051 5965 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:54:06 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.218.61@KRBTEST.COM
Aug 06 10:54:06 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013646, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.61@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM
I20260806 10:54:06.306403 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:06.281820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37242 (local address 127.30.218.61:45205)
0806 10:54:06.282820 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:06.282841 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:06.283242 (+ 401us) server_negotiation.cc:407] Connection header received
0806 10:54:06.284921 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:06.284952 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:06.285521 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:06.286039 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:06.287528 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.289515 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.290937 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.291953 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.292133 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:06.297102 (+ 4969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:06.297115 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:06.297133 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:06.297199 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:06.301384 (+ 4185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:06.302233 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:06.302241 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:06.302249 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:06.302424 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:06.302890 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:06.302894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:06.302896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:06.303561 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:06.303892 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:06.304315 (+ 423us) server_negotiation.cc:299] Negotiation successful
0806 10:54:06.304792 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":424,"threads_started":1}
I20260806 10:54:06.310232 5971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:06.293124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.0.0.1:35602)
0806 10:54:06.293943 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:06.293987 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:06.294154 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:06.294565 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:06.294572 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:06.295277 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:06.295533 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:06.295543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.296951 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.296957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:06.298856 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:06.298876 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.299961 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.299969 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:06.300200 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:06.300882 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:06.300907 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:06.303367 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:06.307404 (+ 4037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:06.307415 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:06.307420 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:06.307945 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:06.308392 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:06.308395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:06.308398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:06.308547 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:06.309191 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:06.309199 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:06.309380 (+ 181us) client_negotiation.cc:769] Sending connection context
0806 10:54:06.309627 (+ 247us) client_negotiation.cc:240] Negotiation successful
0806 10:54:06.309922 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":483,"threads_started":1}
I20260806 10:54:06.310256 5970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:06.293077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35602 (local address 127.30.218.62:34161)
0806 10:54:06.293596 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:06.293603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:06.294085 (+ 482us) server_negotiation.cc:407] Connection header received
0806 10:54:06.294290 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:06.294295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:06.294379 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:06.294475 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:06.295702 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.296775 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.299077 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.299720 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.299831 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:06.303611 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:06.303619 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:06.303623 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:06.303677 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:06.307239 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:06.308105 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:06.308111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:06.308113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:06.308274 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:06.308670 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:06.308674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:06.308676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:06.309046 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:06.309203 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:06.309783 (+ 580us) server_negotiation.cc:299] Negotiation successful
0806 10:54:06.309984 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":247,"threads_started":1}
I20260806 10:54:06.312038 5965 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:54:06 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013646, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.61@KRBTEST.COM for oryx/127.30.218.61@KRBTEST.COM
I20260806 10:54:06.324862 5971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:06.312863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:45205 (local address 127.0.0.1:37246)
0806 10:54:06.313042 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:06.313059 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:06.313159 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:06.313422 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:06.313425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:06.313842 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:06.313985 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:06.313994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.315047 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.315051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:06.315996 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:06.316009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.316674 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.316678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:06.316809 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:06.317419 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:06.317436 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:06.319605 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:06.322488 (+ 2883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:06.322494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:06.322499 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:06.322899 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:06.323231 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:06.323236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:06.323240 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:06.323366 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:06.323904 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:06.323908 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:06.324053 (+ 145us) client_negotiation.cc:769] Sending connection context
0806 10:54:06.324176 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 10:54:06.324479 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 10:54:06.324947 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:06.313060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37246 (local address 127.30.218.61:45205)
0806 10:54:06.313205 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:06.313209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:06.313224 (+ 15us) server_negotiation.cc:407] Connection header received
0806 10:54:06.313265 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:06.313268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:06.313322 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:06.313398 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:06.314104 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.314925 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.316109 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.316521 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.316620 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:06.319805 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:06.319810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:06.319813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:06.319844 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:06.322350 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:06.323010 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:06.323016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:06.323018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:06.323131 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:06.323462 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:06.323467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:06.323470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:06.323796 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:06.323929 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:06.324282 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 10:54:06.324650 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":51}
I20260806 10:54:06.330117 5965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149: Bootstrap starting.
I20260806 10:54:06.333345 5965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:06.334535 5965 log.cc:824] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149: Log is configured to *not* fsync() on all Append() calls
I20260806 10:54:06.337420 5965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149: No bootstrap required, opened a new log
I20260806 10:54:06.344813 5965 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } }
I20260806 10:54:06.345191 5965 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:06.345271 5965 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5fa1e7cf26345d2b7538c6d34ea8149, State: Initialized, Role: FOLLOWER
I20260806 10:54:06.345889 5965 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } }
I20260806 10:54:06.346801 5973 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } } }
I20260806 10:54:06.347131 5973 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:54:06.347898 5965 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 10:54:06.357555 5984 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 10:54:06.357738 5984 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 10:54:06.360391 5897 master_runner.cc:428] Detected that this master f5fa1e7cf26345d2b7538c6d34ea8149 is joining an existing cluster
I20260806 10:54:06.360505 5897 master_runner.cc:432] Initiating AddMaster RPC to add 127.30.218.61:45205
I20260806 10:54:06.390725 5970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:06.367076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35608 (local address 127.30.218.62:34161)
0806 10:54:06.367252 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:06.367257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:06.374219 (+ 6962us) server_negotiation.cc:407] Connection header received
0806 10:54:06.374947 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:06.374954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:06.375079 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:06.375157 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:06.376967 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.378085 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.380320 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.380912 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.381010 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:06.383744 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:06.383751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:06.383755 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:06.383796 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:06.387233 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:06.388014 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:06.388020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:06.388023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:06.388273 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:06.388731 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:06.388737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:06.388740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:06.389133 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:06.389266 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:06.390078 (+ 812us) server_negotiation.cc:299] Negotiation successful
0806 10:54:06.390350 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 10:54:06.392228 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:06.366457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.0.0.1:35608)
0806 10:54:06.374037 (+ 7580us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:06.374068 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:06.374759 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:06.375431 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:06.375436 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:06.376254 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:06.376719 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:06.376779 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.378273 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.378280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:06.380143 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:06.380160 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.381330 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.381336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:06.381529 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:06.382283 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:06.382306 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:06.383495 (+ 1189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:06.387389 (+ 3894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:06.387393 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:06.387397 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:06.387851 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:06.388423 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:06.388427 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:06.388430 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:06.388603 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:06.389402 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:06.389407 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:06.389750 (+ 343us) client_negotiation.cc:769] Sending connection context
0806 10:54:06.391311 (+ 1561us) client_negotiation.cc:240] Negotiation successful
0806 10:54:06.391711 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7373,"thread_start_us":11201,"threads_started":1}
I20260806 10:54:06.393486 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:06.366381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37262 (local address 127.30.218.61:45205)
0806 10:54:06.366616 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:06.366622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:06.378140 (+ 11518us) server_negotiation.cc:407] Connection header received
0806 10:54:06.378377 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:06.378382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:06.378467 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:06.378538 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:06.380239 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.381356 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.383565 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.384287 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.384394 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:06.386771 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:06.386778 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:06.386783 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:06.386817 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:06.390513 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:06.391329 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:06.391335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:06.391338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:06.391490 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:06.391892 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:06.391896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:06.391900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:06.392288 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:06.392438 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:06.392917 (+ 479us) server_negotiation.cc:299] Negotiation successful
0806 10:54:06.393139 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 10:54:06.394677 5990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:06.366789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:45205 (local address 127.0.0.1:37262)
0806 10:54:06.378038 (+ 11249us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:06.378060 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:06.378244 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:06.378755 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:06.378760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:06.379557 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:06.380065 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:06.380081 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.381619 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.381625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:06.383399 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:06.383416 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.384673 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.384680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:06.384852 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:06.385430 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:06.385477 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:06.386572 (+ 1095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:06.390666 (+ 4094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:06.390671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:06.390675 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:06.391201 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:06.391624 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:06.391628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:06.391632 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:06.391793 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:06.392507 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:06.392512 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:06.392657 (+ 145us) client_negotiation.cc:769] Sending connection context
0806 10:54:06.393928 (+ 1271us) client_negotiation.cc:240] Negotiation successful
0806 10:54:06.394235 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11087,"thread_start_us":7465,"threads_started":1}
WARNING: no policy specified for oryx/127.30.218.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.218.60@KRBTEST.COM" created.
Aug 06 10:54:06 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013643, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.62@KRBTEST.COM for oryx/127.30.218.61@KRBTEST.COM
I20260806 10:54:06.416522 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:06.399627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37268 (local address 127.30.218.61:45205)
0806 10:54:06.399788 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:06.399794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:06.400069 (+ 275us) server_negotiation.cc:407] Connection header received
0806 10:54:06.400425 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:06.400430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:06.400512 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:06.400619 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:06.402117 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.403094 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.405042 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.405745 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.405871 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:06.410100 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:06.410107 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:06.410110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:06.410147 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:06.413038 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:06.413907 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:06.413912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:06.413914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:06.414084 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:06.414665 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:06.414669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:06.414671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:06.415034 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:06.415275 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:06.415794 (+ 519us) server_negotiation.cc:299] Negotiation successful
0806 10:54:06.416066 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 10:54:06.416615 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:06.399158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:45205 (local address 127.0.0.1:37268)
0806 10:54:06.399942 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:06.399993 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:06.400223 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:06.400679 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:06.400687 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:06.401615 (+ 928us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:06.401908 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:06.401922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.403248 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.403254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:06.404854 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:06.404872 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.405936 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.405943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:06.406188 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:06.406972 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:06.406999 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:06.409852 (+ 2853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:06.413195 (+ 3343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:06.413212 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:06.413223 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:06.413784 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:06.414203 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:06.414208 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:06.414212 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:06.414537 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:06.415254 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:06.415263 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:06.415461 (+ 198us) client_negotiation.cc:769] Sending connection context
0806 10:54:06.415654 (+ 193us) client_negotiation.cc:240] Negotiation successful
0806 10:54:06.416051 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":518,"threads_started":1}
I20260806 10:54:06.418948 5430 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.30.218.61:45205: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149"
member_type: NON_VOTER
last_known_addr {
host: "127.30.218.61"
port: 45205
}
attrs {
promote: true
}
}
dest_uuid: "4b824ac0a7c94632b4d0aea8540f15bd"
cas_config_opid_index: -1
I20260806 10:54:06.420311 5430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: NON_VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: true } }
I20260806 10:54:06.430536 5996 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f5fa1e7cf26345d2b7538c6d34ea8149 (127.30.218.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: NON_VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: true } } }
I20260806 10:54:06.432579 5939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:06.434680 5996 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.30.218.61:45205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.218.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.60.keytab.
Entry for principal oryx/127.30.218.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.60.keytab.
I20260806 10:54:06.436022 5897 master.cc:537] Master@127.30.218.61:45205 shutting down...
I20260806 10:54:06.435698 5997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: NON_VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: true } } }
I20260806 10:54:06.437182 5997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [sys.catalog]: This master's current role is: LEADER
I20260806 10:54:06.437680 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.60:34453
--webserver_interface=127.30.218.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.60.keytab
--principal=oryx/127.30.218.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:34161,127.30.218.61:45205,127.30.218.60:34453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:54:06.442622 5939 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [term 1 FOLLOWER]: Refusing update from remote peer 4b824ac0a7c94632b4d0aea8540f15bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260806 10:54:06.445776 5996 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: NON_VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 10:54:06.449705 5417 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260806 10:54:06.454257 5897 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:54:06.454804 5897 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:54:06.455096 5897 tablet_replica.cc:330] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149: stopping tablet replica
W20260806 10:54:06.455000 5417 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd -> Peer f5fa1e7cf26345d2b7538c6d34ea8149 (127.30.218.61:45205): Couldn't send request to peer f5fa1e7cf26345d2b7538c6d34ea8149. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 10:54:06.473270 5897 master.cc:552] Master@127.30.218.61:45205 shutdown complete.
I20260806 10:54:06.473440 5897 master_runner.cc:305] Clearing existing system tablet
I20260806 10:54:06.473981 5897 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:06.474092 5897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:06.477672 5897 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260806 10:54:06.479517 6002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:06.479918 5897 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:06.480219 5897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260806 10:54:06.480340 5897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/wal
uuid: "f5fa1e7cf26345d2b7538c6d34ea8149"
format_stamp: "Formatted at 2026-08-06 10:54:06 on dist-test-slave-49h3"
server_key: "7cda9c12a6f455d8b24e08f5a6ec6fc2"
server_key_iv: "1c6f6497be0f93739e433fdf4ab32aa1"
server_key_version: "encryptionkey@0"
I20260806 10:54:06.485731 5897 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:54:06.493266 5897 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 10:54:06.493556 5897 log.cc:1196] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 10:54:06.494037 5897 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149: Deleting consensus metadata
I20260806 10:54:06.494341 5897 master_runner.cc:315] Copying system tablet from 127.30.218.62:34161
I20260806 10:54:06.499711 5897 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149: tablet copy: Beginning tablet copy session from remote peer at address 127.30.218.62:34161
I20260806 10:54:06.517498 5970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:06.500861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35614 (local address 127.30.218.62:34161)
0806 10:54:06.501065 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:06.501072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:06.502041 (+ 969us) server_negotiation.cc:407] Connection header received
0806 10:54:06.502309 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:06.502315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:06.502423 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:06.502586 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:06.503667 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.504823 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.506777 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.507446 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.507583 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:06.510117 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:06.510124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:06.510129 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:06.510173 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:06.513854 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:06.514848 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:06.514855 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:06.514859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:06.515042 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:06.515587 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:06.515593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:06.515596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:06.516013 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:06.516185 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:06.516814 (+ 629us) server_negotiation.cc:299] Negotiation successful
0806 10:54:06.517076 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 10:54:06.517576 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:06.500837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.0.0.1:35614)
0806 10:54:06.501900 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:06.501924 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:06.502094 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:06.502620 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:06.502625 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:06.503167 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:06.503492 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:06.503505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.504983 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.504989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:06.506588 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:06.506604 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.507617 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.507623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:06.507825 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:06.508597 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:06.508622 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:06.509839 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:06.514036 (+ 4197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:06.514043 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:06.514048 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:06.514663 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:06.515199 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:06.515204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:06.515208 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:06.515421 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:06.516163 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:06.516169 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:06.516392 (+ 223us) client_negotiation.cc:769] Sending connection context
0806 10:54:06.516666 (+ 274us) client_negotiation.cc:240] Negotiation successful
0806 10:54:06.517045 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"thread_start_us":760,"threads_started":1}
I20260806 10:54:06.519003 5450 tablet_copy_service.cc:140] P 4b824ac0a7c94632b4d0aea8540f15bd: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f5fa1e7cf26345d2b7538c6d34ea8149 ({username='oryx', principal='oryx/127.30.218.61@KRBTEST.COM'} at 127.0.0.1:35614)
I20260806 10:54:06.519510 5450 tablet_copy_service.cc:161] P 4b824ac0a7c94632b4d0aea8540f15bd: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f5fa1e7cf26345d2b7538c6d34ea8149 at {username='oryx', principal='oryx/127.30.218.61@KRBTEST.COM'} at 127.0.0.1:35614: session id = f5fa1e7cf26345d2b7538c6d34ea8149-00000000000000000000000000000000
I20260806 10:54:06.524446 5450 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 10:54:06.527838 5897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 10:54:06.535873 5897 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149: tablet copy: Starting download of 0 data blocks...
I20260806 10:54:06.536360 5897 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149: tablet copy: Starting download of 1 WAL segments...
I20260806 10:54:06.540983 5897 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 10:54:06.546556 5450 tablet_copy_service.cc:342] P 4b824ac0a7c94632b4d0aea8540f15bd: Request end of tablet copy session f5fa1e7cf26345d2b7538c6d34ea8149-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.30.218.61@KRBTEST.COM'} at 127.0.0.1:35614
I20260806 10:54:06.547004 5450 tablet_copy_service.cc:433] P 4b824ac0a7c94632b4d0aea8540f15bd: ending tablet copy session f5fa1e7cf26345d2b7538c6d34ea8149-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f5fa1e7cf26345d2b7538c6d34ea8149
I20260806 10:54:06.560698 5897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:06.563705 6009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:06.566951 6010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:06.583195 6012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:06.583276 5897 server_base.cc:1034] running on GCE node
I20260806 10:54:06.583880 5897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:06.584784 5897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:06.586140 5897 hybrid_clock.cc:648] HybridClock initialized: now 1786013646586063 us; error 45 us; skew 500 ppm
I20260806 10:54:06.586403 5897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:06 dist-test-slave-49h3 krb5kdc[5384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013646, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:06.588794 5897 init.cc:377] Logged in from keytab as oryx/127.30.218.61@KRBTEST.COM (short username oryx)
I20260806 10:54:06.590253 5897 webserver.cc:460] Webserver started at http://127.30.218.61:38891/ using document root <none> and password file <none>
I20260806 10:54:06.590682 5897 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:06.590786 5897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:06.594179 5897 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 10:54:06.596335 6019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:06.596814 5897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 10:54:06.596989 5897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/wal
uuid: "f5fa1e7cf26345d2b7538c6d34ea8149"
format_stamp: "Formatted at 2026-08-06 10:54:06 on dist-test-slave-49h3"
server_key: "7cda9c12a6f455d8b24e08f5a6ec6fc2"
server_key_iv: "1c6f6497be0f93739e433fdf4ab32aa1"
server_key_version: "encryptionkey@0"
I20260806 10:54:06.597177 5897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:06.626703 5897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:06.631072 5897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:06.657853 5897 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.61:45205
I20260806 10:54:06.657915 6072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.61:45205 every 8 connection(s)
I20260806 10:54:06.659042 5897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-1/data/info.pb
I20260806 10:54:06.665025 6073 sys_catalog.cc:263] Verifying existing consensus state
I20260806 10:54:06.670544 6073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149: Bootstrap starting.
I20260806 10:54:06.693250 6073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 10:54:06.694029 6073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149: Bootstrap complete.
I20260806 10:54:06.695379 6073 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: NON_VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: true } }
I20260806 10:54:06.695725 6073 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f5fa1e7cf26345d2b7538c6d34ea8149, State: Initialized, Role: LEARNER
I20260806 10:54:06.696040 6073 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: NON_VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: true } }
I20260806 10:54:06.697737 6073 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:54:06.697667 6076 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: NON_VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: true } } }
I20260806 10:54:06.698081 6076 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [sys.catalog]: This master's current role is: LEARNER
I20260806 10:54:06.712459 6087 catalog_manager.cc:1259] Loaded cluster ID: fd65f6dd547845159ea0ec495152a496
I20260806 10:54:06.712651 6087 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149: loading cluster ID for follower catalog manager: success
I20260806 10:54:06.715507 6087 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149: acquiring CA information for follower catalog manager: success
I20260806 10:54:06.717921 6087 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 10:54:06.752754 5891 heartbeater.cc:503] Master 127.30.218.62:34161 was elected leader, sending a full tablet report...
W20260806 10:54:06.902690 5999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:06.903133 5999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:06.903281 5999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:06.911957 5999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:54:06.912104 5999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:06.912154 5999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:06.912213 5999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:54:06.912259 5999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:54:06.923103 5999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:34161,127.30.218.61:45205,127.30.218.60:34453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.60.keytab
--principal=oryx/127.30.218.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.60:34453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.218.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:06.924620 5999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:06.926129 5999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:06.941057 6095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:06.941320 6093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:06.941347 6092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:06.942564 5999 server_base.cc:1034] running on GCE node
I20260806 10:54:06.943442 5999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:06.945029 5999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:06.946240 5999 hybrid_clock.cc:648] HybridClock initialized: now 1786013646946188 us; error 48 us; skew 500 ppm
I20260806 10:54:06.946609 5999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:06 dist-test-slave-49h3 krb5kdc[5384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013646, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:06.950110 5999 init.cc:377] Logged in from keytab as oryx/127.30.218.60@KRBTEST.COM (short username oryx)
I20260806 10:54:06.951761 5999 webserver.cc:460] Webserver started at http://127.30.218.60:40589/ using document root <none> and password file <none>
I20260806 10:54:06.952497 5999 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:06.952612 5999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:06.952927 5999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:06.956252 5999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/data/instance:
uuid: "7b65bc5a92ba4967bf17cef40391bb14"
format_stamp: "Formatted at 2026-08-06 10:54:06 on dist-test-slave-49h3"
server_key: "a213c6b3630663c55e9384142bf6b01a"
server_key_iv: "121f837cf71024e763ceb0a2e28658dc"
server_key_version: "encryptionkey@0"
I20260806 10:54:06.957115 5999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/wal/instance:
uuid: "7b65bc5a92ba4967bf17cef40391bb14"
format_stamp: "Formatted at 2026-08-06 10:54:06 on dist-test-slave-49h3"
server_key: "a213c6b3630663c55e9384142bf6b01a"
server_key_iv: "121f837cf71024e763ceb0a2e28658dc"
server_key_version: "encryptionkey@0"
I20260806 10:54:06.963229 5999 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260806 10:54:06.967183 6102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:06.968760 5999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 10:54:06.968955 5999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/wal
uuid: "7b65bc5a92ba4967bf17cef40391bb14"
format_stamp: "Formatted at 2026-08-06 10:54:06 on dist-test-slave-49h3"
server_key: "a213c6b3630663c55e9384142bf6b01a"
server_key_iv: "121f837cf71024e763ceb0a2e28658dc"
server_key_version: "encryptionkey@0"
I20260806 10:54:06.969189 5999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:06.990330 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:06.983755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37282 (local address 127.30.218.61:45205)
0806 10:54:06.984388 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:06.984394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:06.984626 (+ 232us) server_negotiation.cc:407] Connection header received
0806 10:54:06.984801 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:06.984807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:06.984907 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:06.985049 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:06.985523 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.986603 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.988228 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.989178 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.989312 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:06.989685 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:06.989843 (+ 158us) server_negotiation.cc:299] Negotiation successful
0806 10:54:06.989989 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":317,"threads_started":1}
I20260806 10:54:06.990322 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:06.983489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:45205 (local address 127.0.0.1:37282)
0806 10:54:06.984513 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:06.984531 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:06.984680 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:06.985095 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:06.985099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:06.985122 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:06.985312 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:06.985323 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.986777 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.986782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:06.988052 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:06.988070 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:06.989385 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:06.989391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:06.989622 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:06.989711 (+ 89us) client_negotiation.cc:769] Sending connection context
0806 10:54:06.989807 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 10:54:06.989974 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"thread_start_us":808,"threads_started":1}
I20260806 10:54:06.991523 6047 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260806 10:54:06.992107 6076 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b824ac0a7c94632b4d0aea8540f15bd. Latest consensus state: current_term: 1 leader_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: NON_VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: true } } }
I20260806 10:54:06.992360 6076 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [sys.catalog]: This master's current role is: LEARNER
I20260806 10:54:07.016759 5999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:07.020942 5999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:07.021230 5999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:07.040153 5999 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.60:34453
I20260806 10:54:07.040189 6158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.60:34453 every 8 connection(s)
I20260806 10:54:07.041704 5999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/data/info.pb
I20260806 10:54:07.045519 6159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:07.049804 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 5999
I20260806 10:54:07.052920 6159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:54:07 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.218.60@KRBTEST.COM
Aug 06 10:54:07 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013646, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.60@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM
I20260806 10:54:07.072363 6164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.057065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35624 (local address 127.30.218.62:34161)
0806 10:54:07.058165 (+ 1100us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:07.058170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:07.058190 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:54:07.058251 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:07.058256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:07.058327 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:07.058420 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:07.059618 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.060384 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.061850 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.062258 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.062368 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.066491 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:07.066496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:07.066500 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:07.066536 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:07.069316 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.070104 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.070110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.070113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.070267 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.070674 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.070678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.070681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.070937 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:07.071044 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:07.071715 (+ 671us) server_negotiation.cc:299] Negotiation successful
0806 10:54:07.072064 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":972,"thread_start_us":868,"threads_started":1}
I20260806 10:54:07.073527 6163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.056828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.0.0.1:35624)
0806 10:54:07.057532 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:07.057595 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:07.058141 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:07.058514 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:07.058521 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:07.059182 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:07.059460 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:07.059469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.060529 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.060532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:07.061716 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:07.061729 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.062493 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.062500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:07.062652 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.063292 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:07.063329 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:07.066301 (+ 2972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:07.069473 (+ 3172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:07.069486 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:07.069499 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:07.069994 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:07.070367 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:07.070371 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:07.070374 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:07.070577 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:07.071049 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:07.071057 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:07.071312 (+ 255us) client_negotiation.cc:769] Sending connection context
0806 10:54:07.071597 (+ 285us) client_negotiation.cc:240] Negotiation successful
0806 10:54:07.072149 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":104,"thread_start_us":266,"threads_started":1}
I20260806 10:54:07.073545 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.051998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51948 (local address 127.30.218.60:34453)
0806 10:54:07.052832 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:07.052844 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:07.052889 (+ 45us) server_negotiation.cc:407] Connection header received
0806 10:54:07.055037 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:07.055065 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:07.055534 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:07.055915 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:07.057836 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.059274 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.060609 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.061326 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.061531 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.066182 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:07.066196 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:07.066216 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:07.066341 (+ 125us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:07.068954 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.069759 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.069768 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.069772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.069971 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.070355 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.070359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.070362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.070988 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:07.071314 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:07.071605 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 10:54:07.072028 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":603,"spinlock_wait_cycles":22272,"thread_start_us":361,"threads_started":1}
I20260806 10:54:07.075593 6159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:54:07 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013646, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.60@KRBTEST.COM for oryx/127.30.218.61@KRBTEST.COM
I20260806 10:54:07.088476 6163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.076469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:45205 (local address 127.0.0.1:37286)
0806 10:54:07.076625 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:07.076645 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:07.076758 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:07.077073 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:07.077076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:07.077434 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:07.077621 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:07.077629 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.078800 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.078804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:07.079731 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:07.079743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.080381 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.080385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:07.080544 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.081092 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:07.081110 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:07.083240 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:07.086280 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:07.086285 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:07.086288 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:07.086706 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:07.087056 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:07.087060 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:07.087062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:07.087198 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:07.087716 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:07.087719 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:07.087852 (+ 133us) client_negotiation.cc:769] Sending connection context
0806 10:54:07.087959 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 10:54:07.088218 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 10:54:07.088570 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.076384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37286 (local address 127.30.218.61:45205)
0806 10:54:07.076593 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:07.076599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:07.076728 (+ 129us) server_negotiation.cc:407] Connection header received
0806 10:54:07.076862 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:07.076867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:07.076954 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:07.077039 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:07.077776 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.078667 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.079845 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.080270 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.080366 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.083459 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:07.083466 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:07.083469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:07.083507 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:07.086125 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.086817 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.086821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.086823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.086966 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.087299 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.087303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.087305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.087613 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:07.087739 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:07.088093 (+ 354us) server_negotiation.cc:299] Negotiation successful
0806 10:54:07.088323 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 10:54:07.089707 6159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.60" port: 34453 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:54:07 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013646, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.60@KRBTEST.COM for oryx/127.30.218.60@KRBTEST.COM
I20260806 10:54:07.102331 6163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.090359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:34453 (local address 127.0.0.1:51950)
0806 10:54:07.090529 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:07.090544 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:07.090657 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:07.090928 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:07.090932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:07.091313 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:07.091456 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:07.091465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.092396 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.092401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:07.093235 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:07.093246 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.093931 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.093936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:07.094151 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.094721 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:07.094738 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:07.096844 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:07.099913 (+ 3069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:07.099918 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:07.099924 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:07.100295 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:07.100626 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:07.100631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:07.100635 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:07.100794 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:07.101302 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:07.101307 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:07.101437 (+ 130us) client_negotiation.cc:769] Sending connection context
0806 10:54:07.101615 (+ 178us) client_negotiation.cc:240] Negotiation successful
0806 10:54:07.101899 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 10:54:07.102356 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.090551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51950 (local address 127.30.218.60:34453)
0806 10:54:07.090701 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:07.090707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:07.090736 (+ 29us) server_negotiation.cc:407] Connection header received
0806 10:54:07.090775 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:07.090778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:07.090828 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:07.090899 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:07.091589 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.092282 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.093368 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.093797 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.093907 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.097032 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:07.097037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:07.097040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:07.097070 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:07.099773 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.100405 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.100410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.100412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.100528 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.100891 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.100896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.100899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.101195 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:07.101328 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:07.101713 (+ 385us) server_negotiation.cc:299] Negotiation successful
0806 10:54:07.102041 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":54}
I20260806 10:54:07.107617 6159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14: Bootstrap starting.
I20260806 10:54:07.111024 6159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:07.112116 6159 log.cc:824] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14: Log is configured to *not* fsync() on all Append() calls
I20260806 10:54:07.115091 6159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14: No bootstrap required, opened a new log
I20260806 10:54:07.122730 6159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 34453 } }
I20260806 10:54:07.123100 6159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:07.123206 6159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b65bc5a92ba4967bf17cef40391bb14, State: Initialized, Role: FOLLOWER
I20260806 10:54:07.123757 6159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 34453 } }
I20260806 10:54:07.124629 6166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 34453 } } }
I20260806 10:54:07.124961 6166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:54:07.125698 6159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 10:54:07.135214 6178 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 10:54:07.135378 6178 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 10:54:07.139269 5999 master_runner.cc:428] Detected that this master 7b65bc5a92ba4967bf17cef40391bb14 is joining an existing cluster
I20260806 10:54:07.139406 5999 master_runner.cc:432] Initiating AddMaster RPC to add 127.30.218.60:34453
I20260806 10:54:07.161077 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.143866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37302 (local address 127.30.218.61:45205)
0806 10:54:07.144132 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:07.144138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:07.144769 (+ 631us) server_negotiation.cc:407] Connection header received
0806 10:54:07.144954 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:07.144960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:07.145056 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:07.145170 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:07.146631 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.147733 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.149914 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.150586 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.150696 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.153631 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:07.153638 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:07.153642 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:07.153689 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:07.157329 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.158276 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.158282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.158286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.158467 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.159000 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.159005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.159008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.159401 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:07.159543 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:07.160414 (+ 871us) server_negotiation.cc:299] Negotiation successful
0806 10:54:07.160701 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260806 10:54:07.162287 6184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.143814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:45205 (local address 127.0.0.1:37302)
0806 10:54:07.144646 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:07.144665 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:07.144818 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:07.145224 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:07.145229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:07.146013 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:07.146434 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:07.146449 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.147969 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.147978 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:07.149715 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:07.149731 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.151018 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.151025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:07.151239 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.152027 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:07.152051 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:07.153370 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:07.157566 (+ 4196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:07.157575 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:07.157579 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:07.158103 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:07.158626 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:07.158631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:07.158635 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:07.158866 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:07.159634 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:07.159639 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:07.160035 (+ 396us) client_negotiation.cc:769] Sending connection context
0806 10:54:07.161393 (+ 1358us) client_negotiation.cc:240] Negotiation successful
0806 10:54:07.161780 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":1843,"threads_started":1}
I20260806 10:54:07.164124 6164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.143688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35634 (local address 127.30.218.62:34161)
0806 10:54:07.144074 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:07.144081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:07.146149 (+ 2068us) server_negotiation.cc:407] Connection header received
0806 10:54:07.146417 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:07.146422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:07.146508 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:07.146596 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:07.148196 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.149389 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.151723 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.152452 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.152566 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.155648 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:07.155657 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:07.155662 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:07.155715 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:07.159501 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.160670 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.160678 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.160682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.160913 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.161833 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.161839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.161842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.162256 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:07.162707 (+ 451us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:07.163473 (+ 766us) server_negotiation.cc:299] Negotiation successful
0806 10:54:07.163787 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248}
I20260806 10:54:07.165678 6185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.143926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.0.0.1:35634)
0806 10:54:07.146042 (+ 2116us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:07.146060 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:07.146257 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:07.146822 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:07.146827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:07.147610 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:07.147997 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:07.148010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.149631 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.149638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:07.151499 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:07.151517 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.152973 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.152980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:07.153198 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.154048 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:07.154073 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:07.155352 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:07.159735 (+ 4383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:07.159741 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:07.159746 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:07.160476 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:07.161082 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:07.161088 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:07.161092 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:07.161659 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:07.162831 (+ 1172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:07.162836 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:07.163116 (+ 280us) client_negotiation.cc:769] Sending connection context
0806 10:54:07.164670 (+ 1554us) client_negotiation.cc:240] Negotiation successful
0806 10:54:07.165079 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1973,"thread_start_us":408,"threads_started":1}
I20260806 10:54:07.179492 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.146333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51962 (local address 127.30.218.60:34453)
0806 10:54:07.146510 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:07.146514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:07.162637 (+ 16123us) server_negotiation.cc:407] Connection header received
0806 10:54:07.162859 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:07.162864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:07.162944 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:07.163015 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:07.164282 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.165533 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.167514 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.168670 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.168794 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.171858 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:07.171865 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:07.171870 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:07.171908 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:07.175932 (+ 4024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.176891 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.176898 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.176902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.177064 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.177579 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.177584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.177588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.178077 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:07.178229 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:07.178821 (+ 592us) server_negotiation.cc:299] Negotiation successful
0806 10:54:07.179059 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 10:54:07.180423 6184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.146622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:34453 (local address 127.0.0.1:51962)
0806 10:54:07.162543 (+ 15921us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:07.162565 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:07.162727 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:07.163220 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:07.163224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:07.163845 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:07.164123 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:07.164146 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.165843 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.165848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:07.167348 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:07.167366 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.169417 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.169423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:07.169655 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.170341 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:07.170370 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:07.171628 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:07.176150 (+ 4522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:07.176156 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:07.176160 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:07.176708 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:07.177232 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:07.177238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:07.177241 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:07.177406 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:07.178313 (+ 907us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:07.178318 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:07.178497 (+ 179us) client_negotiation.cc:769] Sending connection context
0806 10:54:07.179696 (+ 1199us) client_negotiation.cc:240] Negotiation successful
0806 10:54:07.180015 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15812,"thread_start_us":21500,"threads_started":1}
I20260806 10:54:07.181291 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 5491
Aug 06 10:54:07 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013643, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.62@KRBTEST.COM for oryx/127.30.218.60@KRBTEST.COM
Aug 06 10:54:07 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.218.1@KRBTEST.COM
I20260806 10:54:07.195325 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.172372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:34453 (local address 127.0.0.1:51970)
0806 10:54:07.172720 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:07.172739 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:07.172847 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:07.178405 (+ 5558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:07.178412 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:07.178866 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:07.179049 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:07.179061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.181417 (+ 2356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.181424 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:07.182872 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:07.182893 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.184026 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.184032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:07.184243 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.184943 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:07.184966 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:07.187214 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:07.192157 (+ 4943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:07.192164 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:07.192169 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:07.192795 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:07.193311 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:07.193317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:07.193321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:07.193550 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:07.194267 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:07.194272 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:07.194470 (+ 198us) client_negotiation.cc:769] Sending connection context
0806 10:54:07.194629 (+ 159us) client_negotiation.cc:240] Negotiation successful
0806 10:54:07.194951 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226}
I20260806 10:54:07.195457 6187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.173051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51970 (local address 127.30.218.60:34453)
0806 10:54:07.178010 (+ 4959us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:07.178014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:07.178034 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:54:07.178125 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:07.178128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:07.178199 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:07.178302 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:07.179228 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.181218 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.183087 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.183798 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.183976 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.187487 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:07.187495 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:07.187501 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:07.187544 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:07.191961 (+ 4417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.192958 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.192965 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.192969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.193164 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.193691 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.193697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.193700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.194114 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:07.194295 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:07.194770 (+ 475us) server_negotiation.cc:299] Negotiation successful
0806 10:54:07.195015 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4823,"thread_start_us":4633,"threads_started":1}
I20260806 10:54:07.197057 5430 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.30.218.60:34453: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14"
member_type: NON_VOTER
last_known_addr {
host: "127.30.218.60"
port: 34453
}
attrs {
promote: true
}
}
dest_uuid: "4b824ac0a7c94632b4d0aea8540f15bd"
cas_config_opid_index: 5
I20260806 10:54:07.198220 5430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: NON_VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: true } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: NON_VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: true } }
I20260806 10:54:07.200446 6190 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7b65bc5a92ba4967bf17cef40391bb14 (127.30.218.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: NON_VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: true } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: NON_VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: true } } }
I20260806 10:54:07.201228 6047 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [term 1 LEARNER]: Refusing update from remote peer 4b824ac0a7c94632b4d0aea8540f15bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 10:54:07.202150 6108 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: NON_VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 10:54:07.204577 6190 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.30.218.60:34453
I20260806 10:54:07.205256 5999 master.cc:537] Master@127.30.218.60:34453 shutting down...
I20260806 10:54:07.205838 6108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: NON_VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: true } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: NON_VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: true } } }
I20260806 10:54:07.206172 6108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [sys.catalog]: This master's current role is: LEADER
I20260806 10:54:07.207005 6188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.182544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58716 (local address 127.30.218.1:34939)
0806 10:54:07.183204 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:07.183211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:07.183238 (+ 27us) server_negotiation.cc:407] Connection header received
0806 10:54:07.183292 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:07.183299 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:07.183524 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:07.183676 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:07.185357 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.187099 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.188456 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.189200 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.189338 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.194260 (+ 4922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:07.194281 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:07.194301 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:07.194348 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:07.200919 (+ 6571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.202084 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.202094 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.202098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.202295 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.202943 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.202948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.202951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.203445 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:07.205231 (+ 1786us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:07.205409 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 10:54:07.206549 (+ 1140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":608,"threads_started":1}
I20260806 10:54:07.208482 5615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:07.218088 6047 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7b65bc5a92ba4967bf17cef40391bb14 (127.30.218.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: NON_VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: true } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: NON_VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: true } } }
I20260806 10:54:07.224083 6076 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: NON_VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: true } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: NON_VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: true } } }
I20260806 10:54:07.224480 6076 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [sys.catalog]: This master's current role is: LEARNER
W20260806 10:54:07.224711 5417 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd -> Peer 7b65bc5a92ba4967bf17cef40391bb14 (127.30.218.60:34453): Couldn't send request to peer 7b65bc5a92ba4967bf17cef40391bb14. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 10:54:07.225668 6108 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd: attempting to promote NON_VOTER f5fa1e7cf26345d2b7538c6d34ea8149 to VOTER
I20260806 10:54:07.226730 6108 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: NON_VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: true } }
I20260806 10:54:07.228873 6047 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [term 1 LEARNER]: Refusing update from remote peer 4b824ac0a7c94632b4d0aea8540f15bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 10:54:07.229805 6189 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260806 10:54:07.230321 5417 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd -> Peer 7b65bc5a92ba4967bf17cef40391bb14 (127.30.218.60:34453): Couldn't send request to peer 7b65bc5a92ba4967bf17cef40391bb14. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 10:54:07.232553 6190 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f5fa1e7cf26345d2b7538c6d34ea8149 (127.30.218.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: NON_VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: true } } }
I20260806 10:54:07.233316 6047 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f5fa1e7cf26345d2b7538c6d34ea8149 (127.30.218.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: NON_VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: true } } }
I20260806 10:54:07.235445 6190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: NON_VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: true } } }
I20260806 10:54:07.235723 6190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [sys.catalog]: This master's current role is: LEADER
I20260806 10:54:07.235941 6190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: NON_VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: true } } }
I20260806 10:54:07.236107 6190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [sys.catalog]: This master's current role is: LEADER
I20260806 10:54:07.239682 6076 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: NON_VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: true } } }
I20260806 10:54:07.239845 6166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:54:07.239941 6076 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:54:07.240085 6166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 34453 } }
I20260806 10:54:07.242061 6166 leader_election.cc:288] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b824ac0a7c94632b4d0aea8540f15bd (127.30.218.62:34161), f5fa1e7cf26345d2b7538c6d34ea8149 (127.30.218.61:45205)
I20260806 10:54:07.243350 5440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7b65bc5a92ba4967bf17cef40391bb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" is_pre_election: true
I20260806 10:54:07.244105 6047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7b65bc5a92ba4967bf17cef40391bb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" is_pre_election: true
I20260806 10:54:07.244913 6107 leader_election.cc:302] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b65bc5a92ba4967bf17cef40391bb14; no voters: 4b824ac0a7c94632b4d0aea8540f15bd, f5fa1e7cf26345d2b7538c6d34ea8149
I20260806 10:54:07.245543 6166 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 10:54:07.245959 5999 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260806 10:54:07.246315 5999 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260806 10:54:07.246491 5999 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14: stopping tablet replica
I20260806 10:54:07.264216 5999 master.cc:552] Master@127.30.218.60:34453 shutdown complete.
I20260806 10:54:07.264324 5999 master_runner.cc:305] Clearing existing system tablet
I20260806 10:54:07.264688 5999 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:07.264796 5999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:07.267573 5999 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260806 10:54:07.269043 6196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:07.269487 5999 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:07.269744 5999 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260806 10:54:07.269847 5999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/wal
uuid: "7b65bc5a92ba4967bf17cef40391bb14"
format_stamp: "Formatted at 2026-08-06 10:54:06 on dist-test-slave-49h3"
server_key: "a213c6b3630663c55e9384142bf6b01a"
server_key_iv: "121f837cf71024e763ceb0a2e28658dc"
server_key_version: "encryptionkey@0"
I20260806 10:54:07.273648 5999 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:54:07.279357 5999 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 10:54:07.279549 5999 log.cc:1196] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 10:54:07.279963 5999 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14: Deleting consensus metadata
I20260806 10:54:07.280186 5999 master_runner.cc:315] Copying system tablet from 127.30.218.62:34161
I20260806 10:54:07.287125 5999 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14: tablet copy: Beginning tablet copy session from remote peer at address 127.30.218.62:34161
I20260806 10:54:07.305975 6164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.287940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35648 (local address 127.30.218.62:34161)
0806 10:54:07.288199 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:07.288206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:07.294104 (+ 5898us) server_negotiation.cc:407] Connection header received
0806 10:54:07.294305 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:07.294310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:07.294382 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:07.294472 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:07.295403 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.296174 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.297619 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.298197 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.298337 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.300389 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:07.300395 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:07.300398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:07.300434 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:07.303117 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.303843 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.303848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.303850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.303992 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.304389 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.304393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.304395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.304725 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:07.304866 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:07.305313 (+ 447us) server_negotiation.cc:299] Negotiation successful
0806 10:54:07.305588 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260806 10:54:07.306138 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.287925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.0.0.1:35648)
0806 10:54:07.293964 (+ 6039us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:07.293988 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:07.294163 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:07.294533 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:07.294539 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:07.295020 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:07.295243 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:07.295254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.296373 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.296377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:07.297404 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:07.297419 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.298365 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.298372 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:07.298578 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.299328 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:07.299348 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:07.300179 (+ 831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:07.303282 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:07.303288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:07.303293 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:07.303722 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:07.304110 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:07.304115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:07.304119 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:07.304282 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:07.304849 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:07.304853 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:07.305036 (+ 183us) client_negotiation.cc:769] Sending connection context
0806 10:54:07.305166 (+ 130us) client_negotiation.cc:240] Negotiation successful
0806 10:54:07.305602 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5904,"thread_start_us":5756,"threads_started":1}
I20260806 10:54:07.307052 5450 tablet_copy_service.cc:140] P 4b824ac0a7c94632b4d0aea8540f15bd: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7b65bc5a92ba4967bf17cef40391bb14 ({username='oryx', principal='oryx/127.30.218.60@KRBTEST.COM'} at 127.0.0.1:35648)
I20260806 10:54:07.307235 5450 tablet_copy_service.cc:161] P 4b824ac0a7c94632b4d0aea8540f15bd: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7b65bc5a92ba4967bf17cef40391bb14 at {username='oryx', principal='oryx/127.30.218.60@KRBTEST.COM'} at 127.0.0.1:35648: session id = 7b65bc5a92ba4967bf17cef40391bb14-00000000000000000000000000000000
I20260806 10:54:07.309664 5450 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 10:54:07.311394 5999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 10:54:07.316959 5999 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14: tablet copy: Starting download of 0 data blocks...
I20260806 10:54:07.317281 5999 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14: tablet copy: Starting download of 1 WAL segments...
I20260806 10:54:07.319598 5999 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 10:54:07.323133 5450 tablet_copy_service.cc:342] P 4b824ac0a7c94632b4d0aea8540f15bd: Request end of tablet copy session 7b65bc5a92ba4967bf17cef40391bb14-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.30.218.60@KRBTEST.COM'} at 127.0.0.1:35648
I20260806 10:54:07.323285 5450 tablet_copy_service.cc:433] P 4b824ac0a7c94632b4d0aea8540f15bd: ending tablet copy session 7b65bc5a92ba4967bf17cef40391bb14-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7b65bc5a92ba4967bf17cef40391bb14
I20260806 10:54:07.334503 5999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:07.337064 6203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:07.343381 6204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:07.347162 6206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:07.348122 5999 server_base.cc:1034] running on GCE node
I20260806 10:54:07.348472 5999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:07.349123 5999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:07.350324 5999 hybrid_clock.cc:648] HybridClock initialized: now 1786013647350259 us; error 52 us; skew 500 ppm
I20260806 10:54:07.350559 5999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:07 dist-test-slave-49h3 krb5kdc[5384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013647, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:07.352422 5999 init.cc:377] Logged in from keytab as oryx/127.30.218.60@KRBTEST.COM (short username oryx)
I20260806 10:54:07.358062 5999 webserver.cc:460] Webserver started at http://127.30.218.60:46835/ using document root <none> and password file <none>
I20260806 10:54:07.358425 5999 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:07.358525 5999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:07.360996 5999 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 10:54:07.362696 6214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:07.363111 5999 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 10:54:07.363246 5999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/wal
uuid: "7b65bc5a92ba4967bf17cef40391bb14"
format_stamp: "Formatted at 2026-08-06 10:54:06 on dist-test-slave-49h3"
server_key: "a213c6b3630663c55e9384142bf6b01a"
server_key_iv: "121f837cf71024e763ceb0a2e28658dc"
server_key_version: "encryptionkey@0"
I20260806 10:54:07.363417 5999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:07.375365 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 5491
I20260806 10:54:07.376160 5999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:07.384588 5999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:07.400717 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:34939
--local_ip_for_outbound_sockets=127.30.218.1
--tserver_master_addrs=127.30.218.62:34161
--webserver_port=38781
--webserver_interface=127.30.218.1
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.1.keytab
--principal=oryx/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.30.218.62:34161,127.30.218.61:45205
--tserver_master_addrs=127.30.218.62:34161,127.30.218.61:45205,127.30.218.60:34453 with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:54:07.408846 5999 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.60:34453
I20260806 10:54:07.408877 6266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.60:34453 every 8 connection(s)
I20260806 10:54:07.409855 5999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/master-2/data/info.pb
I20260806 10:54:07.415720 6267 sys_catalog.cc:263] Verifying existing consensus state
I20260806 10:54:07.418773 6267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14: Bootstrap starting.
I20260806 10:54:07.434362 6267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 10:54:07.434852 6267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14: Bootstrap complete.
I20260806 10:54:07.435760 6267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: NON_VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: true } }
I20260806 10:54:07.436012 6267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7b65bc5a92ba4967bf17cef40391bb14, State: Initialized, Role: LEARNER
I20260806 10:54:07.436262 6267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: NON_VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: true } }
I20260806 10:54:07.437052 6271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: NON_VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: true } } }
I20260806 10:54:07.437377 6267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:54:07.437595 6271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [sys.catalog]: This master's current role is: LEARNER
I20260806 10:54:07.446115 6282 catalog_manager.cc:1259] Loaded cluster ID: fd65f6dd547845159ea0ec495152a496
I20260806 10:54:07.446219 6282 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14: loading cluster ID for follower catalog manager: success
I20260806 10:54:07.448812 6282 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14: acquiring CA information for follower catalog manager: success
I20260806 10:54:07.450909 6282 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 10:54:07.792268 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.785896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:34453 (local address 127.0.0.1:51984)
0806 10:54:07.786428 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:07.786454 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:07.786634 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:07.787088 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:07.787093 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:07.787123 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:07.787350 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:07.787361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.788661 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.788665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:07.790095 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:07.790113 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.791390 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.791396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:07.791624 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.791657 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 10:54:07.791748 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 10:54:07.791911 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":293,"threads_started":1}
I20260806 10:54:07.792269 6285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.786057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51984 (local address 127.30.218.60:34453)
0806 10:54:07.786764 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:07.786771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:07.786787 (+ 16us) server_negotiation.cc:407] Connection header received
0806 10:54:07.786854 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:07.786857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:07.786931 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:07.787042 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:07.787499 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.788499 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.790265 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.791199 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.791334 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.791570 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:07.791797 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 10:54:07.791941 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":405,"threads_started":1}
I20260806 10:54:07.793614 6241 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260806 10:54:07.794289 6271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b824ac0a7c94632b4d0aea8540f15bd. Latest consensus state: current_term: 1 leader_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: NON_VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: true } } }
I20260806 10:54:07.794610 6271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [sys.catalog]: This master's current role is: LEARNER
W20260806 10:54:07.814538 6269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:07.814858 6269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:07.814957 6269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:07.822841 6269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:07.822995 6269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:07.823117 6269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:54:07.833887 6269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.1.keytab
--principal=oryx/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:34939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=38781
--tserver_master_addrs=127.30.218.62:34161,127.30.218.61:45205,127.30.218.60:34453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:07.835322 6269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:07.836695 6269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:07.851366 6291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:07.851338 6290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:07.851308 6293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:07.853029 6269 server_base.cc:1034] running on GCE node
I20260806 10:54:07.853799 6269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:07.854862 6269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:07.856086 6269 hybrid_clock.cc:648] HybridClock initialized: now 1786013647856015 us; error 61 us; skew 500 ppm
I20260806 10:54:07.856434 6269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:07 dist-test-slave-49h3 krb5kdc[5384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013647, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:07.860003 6269 init.cc:377] Logged in from keytab as oryx/127.30.218.1@KRBTEST.COM (short username oryx)
I20260806 10:54:07.861835 6269 webserver.cc:460] Webserver started at http://127.30.218.1:38781/ using document root <none> and password file <none>
I20260806 10:54:07.862679 6269 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:07.862840 6269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:07.869731 6269 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 10:54:07.872920 6300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:07.874555 6269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:54:07.874754 6269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "3478ba34809347f3950b6b5326dd157d"
format_stamp: "Formatted at 2026-08-06 10:54:04 on dist-test-slave-49h3"
server_key: "e30a56dd22389286f85d08211a97e265"
server_key_iv: "6388040811b2f1b18004d72e96a8699b"
server_key_version: "encryptionkey@0"
I20260806 10:54:07.875953 6269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:07.890154 6269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:07.894903 6269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:07.895222 6269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:07.896334 6269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:54:07.898077 6269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:07.898181 6269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:07.898299 6269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:07.898437 6269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260806 10:54:07.932802 6269 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:34939
I20260806 10:54:07.932853 6415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:34939 every 8 connection(s)
I20260806 10:54:07.934485 6269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:54:07.946473 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 6269
I20260806 10:54:07.947032 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 5626
Aug 06 10:54:07 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013647, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.60@KRBTEST.COM
Aug 06 10:54:07 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013647, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM
Aug 06 10:54:07 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013647, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.61@KRBTEST.COM
Aug 06 10:54:07 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.218.2@KRBTEST.COM
I20260806 10:54:07.981657 6285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.945549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:39489 (local address 127.30.218.60:34453)
0806 10:54:07.945834 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:07.945845 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:07.945866 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:54:07.945969 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:07.945975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:07.946145 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:07.946330 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:07.958826 (+ 12496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.960135 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.964474 (+ 4339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.965291 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.965412 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.972541 (+ 7129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:07.972549 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:07.972554 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:07.972602 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:07.976107 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.977732 (+ 1625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.977740 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.977744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.977947 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.978993 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.979000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.979003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.979406 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:07.979563 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:07.980951 (+ 1388us) server_negotiation.cc:299] Negotiation successful
0806 10:54:07.981267 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 10:54:07.983476 6427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.951528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:49105 (local address 127.30.218.61:45205)
0806 10:54:07.961949 (+ 10421us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:07.961956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:07.967101 (+ 5145us) server_negotiation.cc:407] Connection header received
0806 10:54:07.967351 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:07.967358 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:07.967462 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:07.967576 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:07.968586 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.969749 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.971699 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.972401 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.972542 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.976503 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:07.976510 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:07.976515 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:07.976569 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:07.979880 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.980960 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.980966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.980969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.981137 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.981663 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.981668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.981670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.981992 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:07.982131 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:07.982860 (+ 729us) server_negotiation.cc:299] Negotiation successful
0806 10:54:07.983095 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10269,"thread_start_us":10129,"threads_started":1}
I20260806 10:54:07.983902 6422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.948978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:47331 (local address 127.30.218.62:34161)
0806 10:54:07.962401 (+ 13423us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:07.962409 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:07.964750 (+ 2341us) server_negotiation.cc:407] Connection header received
0806 10:54:07.964831 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:07.964837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:07.964928 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:07.965050 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:07.967087 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.968202 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.970573 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.971238 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.971343 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.976249 (+ 4906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:07.976256 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:07.976261 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:07.976308 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:07.979804 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.981015 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.981032 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.981035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.981198 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.981629 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.981634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.981637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.982014 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:07.982146 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:07.983251 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0806 10:54:07.983500 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13289,"thread_start_us":13163,"threads_started":1}
I20260806 10:54:07.984308 6418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.938200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:34453 (local address 127.30.218.1:39489)
0806 10:54:07.942188 (+ 3988us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:07.942263 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:07.944905 (+ 2642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:07.947104 (+ 2199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:07.947148 (+ 44us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:07.955066 (+ 7918us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:07.956319 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:07.956360 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.960565 (+ 4205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.960573 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:07.963268 (+ 2695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:07.963294 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.966041 (+ 2747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.966050 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:07.966524 (+ 474us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.968527 (+ 2003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:07.968572 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:07.972207 (+ 3635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:07.976340 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:07.976351 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:07.976374 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:07.977560 (+ 1186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:07.978504 (+ 944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:07.978509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:07.978513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:07.978836 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:07.979677 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:07.979686 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:07.980399 (+ 713us) client_negotiation.cc:769] Sending connection context
0806 10:54:07.982028 (+ 1629us) client_negotiation.cc:240] Negotiation successful
0806 10:54:07.982724 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3617,"thread_start_us":3367,"threads_started":1}
I20260806 10:54:07.984810 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.951043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:45205 (local address 127.30.218.1:49105)
0806 10:54:07.966975 (+ 15932us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:07.966995 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:07.967212 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:07.967608 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:07.967612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:07.968107 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:07.968405 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:07.968416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.969933 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.969939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:07.971508 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:07.971538 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.972839 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.972846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:07.973041 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.973697 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:07.973724 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:07.976276 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:07.980017 (+ 3741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:07.980021 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:07.980024 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:07.980803 (+ 779us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:07.981274 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:07.981280 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:07.981284 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:07.981512 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:07.982216 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:07.982221 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:07.982439 (+ 218us) client_negotiation.cc:769] Sending connection context
0806 10:54:07.982602 (+ 163us) client_negotiation.cc:240] Negotiation successful
0806 10:54:07.984284 (+ 1682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15798,"mutex_wait_us":9,"spinlock_wait_cycles":94592,"thread_start_us":12994,"threads_started":1}
I20260806 10:54:07.985891 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.948822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.30.218.1:47331)
0806 10:54:07.964433 (+ 15611us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:07.964457 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:07.964615 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:07.965261 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:07.965266 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:07.965982 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:07.966415 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:07.966429 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.968594 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.968599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:07.970392 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:07.970410 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.971666 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.971672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:07.971827 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.972528 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:07.972551 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:07.975967 (+ 3416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:07.979975 (+ 4008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:07.979980 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:07.979984 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:07.980878 (+ 894us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:07.981336 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:07.981341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:07.981344 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:07.981494 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:07.982216 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:07.982221 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:07.982439 (+ 218us) client_negotiation.cc:769] Sending connection context
0806 10:54:07.982602 (+ 163us) client_negotiation.cc:240] Negotiation successful
0806 10:54:07.982913 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15453,"mutex_wait_us":28,"spinlock_wait_cycles":28672,"thread_start_us":17795,"threads_started":1}
I20260806 10:54:07.986611 6419 heartbeater.cc:347] Connected to a master server at 127.30.218.61:45205
I20260806 10:54:07.986722 6416 heartbeater.cc:347] Connected to a master server at 127.30.218.60:34453
I20260806 10:54:07.986985 6419 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:07.987063 6417 heartbeater.cc:347] Connected to a master server at 127.30.218.62:34161
I20260806 10:54:07.987180 6417 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:07.986991 6416 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:07.987609 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:07.948565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35798 (local address 127.30.218.2:42745)
0806 10:54:07.962933 (+ 14368us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:07.962940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:07.962967 (+ 27us) server_negotiation.cc:407] Connection header received
0806 10:54:07.963043 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:07.963049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:07.963321 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:07.963601 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:07.966112 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.967615 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.969927 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:07.970901 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:07.972213 (+ 1312us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:07.977975 (+ 5762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:07.977997 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:07.978019 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:07.978070 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:07.982622 (+ 4552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.983748 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.983757 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.983760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.983941 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:07.984438 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:07.984444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:07.984448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:07.984906 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:07.985213 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:07.986852 (+ 1639us) server_negotiation.cc:299] Negotiation successful
0806 10:54:07.987176 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14193,"thread_start_us":14021,"threads_started":1}
I20260806 10:54:07.987941 6419 heartbeater.cc:511] Master 127.30.218.61:45205 requested a full tablet report, sending...
I20260806 10:54:07.988049 6416 heartbeater.cc:511] Master 127.30.218.60:34453 requested a full tablet report, sending...
I20260806 10:54:07.989094 6417 heartbeater.cc:511] Master 127.30.218.62:34161 requested a full tablet report, sending...
I20260806 10:54:07.989286 5749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:07.990834 6037 ts_manager.cc:194] Registered new tserver with Master: 3478ba34809347f3950b6b5326dd157d (127.30.218.1:34939)
I20260806 10:54:07.990898 6231 ts_manager.cc:194] Registered new tserver with Master: 3478ba34809347f3950b6b5326dd157d (127.30.218.1:34939)
I20260806 10:54:07.992064 5430 ts_manager.cc:194] Re-registered known tserver with Master: 3478ba34809347f3950b6b5326dd157d (127.30.218.1:34939)
I20260806 10:54:07.992969 5430 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.218.1@KRBTEST.COM'} at 127.30.218.1:47331
I20260806 10:54:08.076658 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 5626
I20260806 10:54:08.093426 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:42745
--local_ip_for_outbound_sockets=127.30.218.2
--tserver_master_addrs=127.30.218.62:34161
--webserver_port=36785
--webserver_interface=127.30.218.2
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.2.keytab
--principal=oryx/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.30.218.62:34161,127.30.218.61:45205
--tserver_master_addrs=127.30.218.62:34161,127.30.218.61:45205,127.30.218.60:34453 with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:54:08.303989 6283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: NON_VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: true } } }
I20260806 10:54:08.304405 6283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [sys.catalog]: This master's current role is: LEADER
I20260806 10:54:08.306404 6283 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd: attempting to promote NON_VOTER 7b65bc5a92ba4967bf17cef40391bb14 to VOTER
I20260806 10:54:08.307083 6283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: false } }
I20260806 10:54:08.320013 6047 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [term 1 FOLLOWER]: Refusing update from remote peer 4b824ac0a7c94632b4d0aea8540f15bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260806 10:54:08.320181 6241 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [term 1 LEARNER]: Refusing update from remote peer 4b824ac0a7c94632b4d0aea8540f15bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260806 10:54:08.320863 6431 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 10:54:08.321403 6283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 10:54:08.325857 6283 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7b65bc5a92ba4967bf17cef40391bb14 (127.30.218.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: false } } }
I20260806 10:54:08.326763 6047 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7b65bc5a92ba4967bf17cef40391bb14 (127.30.218.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: false } } }
I20260806 10:54:08.326956 6241 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7b65bc5a92ba4967bf17cef40391bb14 (127.30.218.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: false } } }
I20260806 10:54:08.330600 6283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: false } } }
I20260806 10:54:08.330904 6283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [sys.catalog]: This master's current role is: LEADER
I20260806 10:54:08.330545 6430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: false } } }
I20260806 10:54:08.331005 6430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [sys.catalog]: This master's current role is: LEADER
I20260806 10:54:08.331173 6283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: false } } }
I20260806 10:54:08.331377 6283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd [sys.catalog]: This master's current role is: LEADER
I20260806 10:54:08.334285 6437 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: false } } }
I20260806 10:54:08.335778 6438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: false } } }
I20260806 10:54:08.336359 6438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:54:08.336576 6437 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 10:54:08.459182 6429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:08.459509 6429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:08.459652 6429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:08.467537 6429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:08.467684 6429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:08.467832 6429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:54:08.478554 6429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.2.keytab
--principal=oryx/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:42745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=36785
--tserver_master_addrs=127.30.218.62:34161,127.30.218.61:45205,127.30.218.60:34453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:08.480072 6429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:08.481951 6429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:08.497797 6443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:08.497732 6446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:08.497802 6444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:08.498443 6429 server_base.cc:1034] running on GCE node
I20260806 10:54:08.499356 6429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:08.500488 6429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:08.501693 6429 hybrid_clock.cc:648] HybridClock initialized: now 1786013648501630 us; error 67 us; skew 500 ppm
I20260806 10:54:08.502024 6429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:08 dist-test-slave-49h3 krb5kdc[5384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013648, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:08.505579 6429 init.cc:377] Logged in from keytab as oryx/127.30.218.2@KRBTEST.COM (short username oryx)
I20260806 10:54:08.507397 6429 webserver.cc:460] Webserver started at http://127.30.218.2:36785/ using document root <none> and password file <none>
I20260806 10:54:08.508214 6429 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:08.508350 6429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:08.514652 6429 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 10:54:08.517793 6453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:08.519426 6429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:54:08.519627 6429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "ef6b396ce9db43c49069415d3b6881df"
format_stamp: "Formatted at 2026-08-06 10:54:05 on dist-test-slave-49h3"
server_key: "4e09613aee4fc4024e5d2ea53182f2e6"
server_key_iv: "7af5f3c6841b397a74a91d42939f3239"
server_key_version: "encryptionkey@0"
I20260806 10:54:08.520756 6429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:08.534809 6429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:08.539630 6429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:08.539935 6429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:08.541003 6429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:54:08.542536 6429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:08.542629 6429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:08.542717 6429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:08.542771 6429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:08.571873 6429 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:42745
I20260806 10:54:08.571907 6568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:42745 every 8 connection(s)
I20260806 10:54:08.573299 6429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:54:08.582444 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 6429
I20260806 10:54:08.582799 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 5761
Aug 06 10:54:08 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013648, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.61@KRBTEST.COM
I20260806 10:54:08.605962 6573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:08.576057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:58141 (local address 127.30.218.61:45205)
0806 10:54:08.576505 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:08.576510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:08.582043 (+ 5533us) server_negotiation.cc:407] Connection header received
0806 10:54:08.582211 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:08.582215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:08.582290 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:08.582383 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:08.585286 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:08.586791 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:08.590093 (+ 3302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:08.590878 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:08.590996 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:08.596835 (+ 5839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:08.596845 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:08.596852 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:08.596906 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:08.601076 (+ 4170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:08.602423 (+ 1347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:08.602431 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:08.602435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:08.602636 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:08.603352 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:08.603359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:08.603362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:08.603807 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:08.603972 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:08.605276 (+ 1304us) server_negotiation.cc:299] Negotiation successful
0806 10:54:08.605582 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":234,"threads_started":1}
I20260806 10:54:08.608556 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:08.576695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:45205 (local address 127.30.218.2:58141)
0806 10:54:08.581925 (+ 5230us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:08.581945 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:08.582082 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:08.582713 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:08.582727 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:08.583847 (+ 1120us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:08.584978 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:08.585010 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:08.587118 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:08.587124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:08.589839 (+ 2715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:08.589863 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:08.591501 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:08.591510 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:08.591924 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:08.592854 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:08.592893 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:08.596416 (+ 3523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:08.601339 (+ 4923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:08.601352 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:08.601378 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:08.602239 (+ 861us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:08.602834 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:08.602840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:08.602844 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:08.603199 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:08.604091 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:08.604102 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:08.604709 (+ 607us) client_negotiation.cc:769] Sending connection context
0806 10:54:08.606344 (+ 1635us) client_negotiation.cc:240] Negotiation successful
0806 10:54:08.606996 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5062,"mutex_wait_us":29,"spinlock_wait_cycles":58752,"thread_start_us":4771,"threads_started":1}
I20260806 10:54:08.610672 6571 heartbeater.cc:347] Connected to a master server at 127.30.218.61:45205
I20260806 10:54:08.610991 6571 heartbeater.cc:464] Registering TS with master...
Aug 06 10:54:08 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013648, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.60@KRBTEST.COM
I20260806 10:54:08.612035 6571 heartbeater.cc:511] Master 127.30.218.61:45205 requested a full tablet report, sending...
I20260806 10:54:08.614183 6037 ts_manager.cc:194] Registered new tserver with Master: ef6b396ce9db43c49069415d3b6881df (127.30.218.2:42745)
Aug 06 10:54:08 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013648, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM
Aug 06 10:54:08 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.218.3@KRBTEST.COM
I20260806 10:54:08.622284 6575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:08.576695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.30.218.2:57501)
0806 10:54:08.577541 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:08.577579 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:08.579330 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:08.605374 (+ 26044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:08.605378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:08.605939 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:08.606337 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:08.606351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:08.608576 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:08.608582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:08.610561 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:08.610580 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:08.612522 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:08.612529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:08.612722 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:08.613400 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:08.613425 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:08.615882 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:08.619364 (+ 3482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:08.619371 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:08.619375 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:08.619785 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:08.620236 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:08.620241 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:08.620245 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:08.620441 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:08.621085 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:08.621090 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:08.621256 (+ 166us) client_negotiation.cc:769] Sending connection context
0806 10:54:08.621415 (+ 159us) client_negotiation.cc:240] Negotiation successful
0806 10:54:08.621825 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":21,"thread_start_us":357,"threads_started":1}
I20260806 10:54:08.622562 6577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:08.582322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:37779 (local address 127.30.218.60:34453)
0806 10:54:08.601870 (+ 19548us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:08.601878 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:08.601902 (+ 24us) server_negotiation.cc:407] Connection header received
0806 10:54:08.601973 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:08.601980 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:08.602080 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:08.602200 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:08.603813 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:08.605014 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:08.607097 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:08.607722 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:08.607853 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:08.615003 (+ 7150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:08.615011 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:08.615016 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:08.615062 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:08.619685 (+ 4623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:08.620386 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:08.620393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:08.620396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:08.620562 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:08.620978 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:08.620983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:08.620987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:08.621337 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:08.621514 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:08.621952 (+ 438us) server_negotiation.cc:299] Negotiation successful
0806 10:54:08.622204 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19366,"thread_start_us":21292,"threads_started":1}
I20260806 10:54:08.622879 6580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:08.585980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:34453 (local address 127.30.218.2:37779)
0806 10:54:08.598567 (+ 12587us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:08.598592 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:08.598751 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:08.602443 (+ 3692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:08.602449 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:08.603088 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:08.603482 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:08.603496 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:08.605210 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:08.605216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:08.606893 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:08.606909 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:08.608163 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:08.608170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:08.608376 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:08.609716 (+ 1340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:08.609740 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:08.614710 (+ 4970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:08.619818 (+ 5108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:08.619823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:08.619826 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:08.620238 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:08.620703 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:08.620708 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:08.620712 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:08.620848 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:08.621485 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:08.621491 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:08.621664 (+ 173us) client_negotiation.cc:769] Sending connection context
0806 10:54:08.621783 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 10:54:08.622069 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12421,"mutex_wait_us":38,"thread_start_us":12821,"threads_started":1}
I20260806 10:54:08.623745 6569 heartbeater.cc:347] Connected to a master server at 127.30.218.60:34453
I20260806 10:54:08.623869 6569 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:08.623865 6574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:08.576072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:57501 (local address 127.30.218.62:34161)
0806 10:54:08.604994 (+ 28922us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:08.605000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:08.605021 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:54:08.605086 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:08.605091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:08.605200 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:08.605335 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:08.607269 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:08.608387 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:08.611621 (+ 3234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:08.612315 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:08.612441 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:08.616167 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:08.616175 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:08.616179 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:08.616237 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:08.619213 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:08.619912 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:08.619918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:08.619922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:08.620106 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:08.620558 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:08.620562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:08.620564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:08.620957 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:08.621118 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:08.623234 (+ 2116us) server_negotiation.cc:299] Negotiation successful
0806 10:54:08.623495 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28766,"thread_start_us":29397,"threads_started":1}
I20260806 10:54:08.624295 6569 heartbeater.cc:511] Master 127.30.218.60:34453 requested a full tablet report, sending...
I20260806 10:54:08.624789 6570 heartbeater.cc:347] Connected to a master server at 127.30.218.62:34161
I20260806 10:54:08.624931 6570 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:08.625386 6570 heartbeater.cc:511] Master 127.30.218.62:34161 requested a full tablet report, sending...
I20260806 10:54:08.625406 6231 ts_manager.cc:194] Registered new tserver with Master: ef6b396ce9db43c49069415d3b6881df (127.30.218.2:42745)
I20260806 10:54:08.626467 5430 ts_manager.cc:194] Re-registered known tserver with Master: ef6b396ce9db43c49069415d3b6881df (127.30.218.2:42745)
I20260806 10:54:08.627329 5430 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.218.2@KRBTEST.COM'} at 127.30.218.2:57501
I20260806 10:54:08.629487 6579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:08.583988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53656 (local address 127.30.218.3:36483)
0806 10:54:08.604371 (+ 20383us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:08.604382 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:08.606440 (+ 2058us) server_negotiation.cc:407] Connection header received
0806 10:54:08.606589 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:08.606601 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:08.607031 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:08.607182 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:08.609641 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:08.611132 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:08.613021 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:08.613951 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:08.614084 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:08.620810 (+ 6726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:08.620831 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:08.620852 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:08.620907 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:08.624661 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:08.625697 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:08.625708 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:08.625711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:08.625951 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:08.627712 (+ 1761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:08.627717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:08.627720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:08.628171 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:08.628477 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:08.628827 (+ 350us) server_negotiation.cc:299] Negotiation successful
0806 10:54:08.629107 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20178,"thread_start_us":20398,"threads_started":1}
I20260806 10:54:08.630694 5886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:08.718441 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 5761
I20260806 10:54:08.735827 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:36483
--local_ip_for_outbound_sockets=127.30.218.3
--tserver_master_addrs=127.30.218.62:34161
--webserver_port=39709
--webserver_interface=127.30.218.3
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.3.keytab
--principal=oryx/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.30.218.62:34161,127.30.218.61:45205
--tserver_master_addrs=127.30.218.62:34161,127.30.218.61:45205,127.30.218.60:34453 with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:54:08.996197 6417 heartbeater.cc:503] Master 127.30.218.62:34161 was elected leader, sending a full tablet report...
W20260806 10:54:09.095557 6582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:09.095878 6582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:09.095984 6582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:09.103467 6582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:09.103593 6582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:09.103717 6582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:54:09.114007 6582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.3.keytab
--principal=oryx/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:36483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=39709
--tserver_master_addrs=127.30.218.62:34161,127.30.218.61:45205,127.30.218.60:34453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:09.115368 6582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:09.116673 6582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:09.129978 6588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:09.129946 6589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:09.130010 6591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:09.131518 6582 server_base.cc:1034] running on GCE node
I20260806 10:54:09.132169 6582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:09.133181 6582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:09.134399 6582 hybrid_clock.cc:648] HybridClock initialized: now 1786013649134344 us; error 47 us; skew 500 ppm
I20260806 10:54:09.134722 6582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:09 dist-test-slave-49h3 krb5kdc[5384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013649, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:09.137931 6582 init.cc:377] Logged in from keytab as oryx/127.30.218.3@KRBTEST.COM (short username oryx)
I20260806 10:54:09.139490 6582 webserver.cc:460] Webserver started at http://127.30.218.3:39709/ using document root <none> and password file <none>
I20260806 10:54:09.140170 6582 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:09.140309 6582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:09.146193 6582 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 10:54:09.149168 6598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:09.150697 6582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:54:09.150879 6582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "b9d816dd0f9147479f65e57e4a4f33ba"
format_stamp: "Formatted at 2026-08-06 10:54:05 on dist-test-slave-49h3"
server_key: "c28b6f9cb18fd50a26111e6748a905c5"
server_key_iv: "e68a00c14b21b7fd7cf5fcc6e7ed34d8"
server_key_version: "encryptionkey@0"
I20260806 10:54:09.151947 6582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:09.175773 6582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:09.180307 6582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:09.180622 6582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:09.181792 6582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:54:09.183390 6582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:09.183498 6582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:09.183609 6582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:09.183689 6582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:09.216768 6582 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:36483
I20260806 10:54:09.216827 6712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:36483 every 8 connection(s)
I20260806 10:54:09.218407 6582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:54:09.225260 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 6582
Aug 06 10:54:09 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013649, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.61@KRBTEST.COM
Aug 06 10:54:09 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013649, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM
Aug 06 10:54:09 dist-test-slave-49h3 krb5kdc[5384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013649, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.60@KRBTEST.COM
I20260806 10:54:09.276049 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:09.243275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:35295 (local address 127.30.218.60:34453)
0806 10:54:09.244835 (+ 1560us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:09.244842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:09.248429 (+ 3587us) server_negotiation.cc:407] Connection header received
0806 10:54:09.248934 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:09.248941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:09.249054 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:09.249141 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:09.250715 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.251887 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.254132 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.254799 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.254898 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:09.264076 (+ 9178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:09.264083 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:09.264088 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:09.264127 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:09.268170 (+ 4043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:09.269412 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:09.269419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:09.269423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:09.269635 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:09.273356 (+ 3721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:09.273363 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:09.273367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:09.273855 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:09.274030 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:09.275254 (+ 1224us) server_negotiation.cc:299] Negotiation successful
0806 10:54:09.275551 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1442,"thread_start_us":1322,"threads_started":1}
I20260806 10:54:09.278615 6719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:09.222572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.30.218.3:47299)
0806 10:54:09.232696 (+ 10124us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:09.232773 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:09.235602 (+ 2829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:09.242578 (+ 6976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:09.242589 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:09.243556 (+ 967us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:09.244349 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:09.244370 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.255143 (+ 10773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.255150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:09.256983 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:09.257004 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.258384 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.258391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:09.258561 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:09.259194 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:09.259218 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:09.262739 (+ 3521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:09.271447 (+ 8708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:09.271453 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:09.271456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:09.272200 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:09.274968 (+ 2768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:09.274974 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:09.274978 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:09.275141 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:09.275916 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:09.275921 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:09.276104 (+ 183us) client_negotiation.cc:769] Sending connection context
0806 10:54:09.276283 (+ 179us) client_negotiation.cc:240] Negotiation successful
0806 10:54:09.276985 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9669,"thread_start_us":13448,"threads_started":1}
I20260806 10:54:09.279552 6723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:09.243166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:34453 (local address 127.30.218.3:35295)
0806 10:54:09.248308 (+ 5142us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:09.248334 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:09.248764 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:09.249367 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:09.249373 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:09.250029 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:09.250511 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:09.250526 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.252193 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.252198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:09.253951 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:09.253969 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.257356 (+ 3387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.257362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:09.257584 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:09.258787 (+ 1203us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:09.258833 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:09.263807 (+ 4974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:09.268381 (+ 4574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:09.268393 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:09.268416 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:09.269239 (+ 823us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:09.270724 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:09.270729 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:09.270733 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:09.271084 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:09.274144 (+ 3060us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:09.274155 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:09.274710 (+ 555us) client_negotiation.cc:769] Sending connection context
0806 10:54:09.276478 (+ 1768us) client_negotiation.cc:240] Negotiation successful
0806 10:54:09.279086 (+ 2608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4978,"mutex_wait_us":2031,"thread_start_us":5290,"threads_started":1}
I20260806 10:54:09.281054 6713 heartbeater.cc:347] Connected to a master server at 127.30.218.60:34453
I20260806 10:54:09.281330 6717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:09.221810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:49203 (local address 127.30.218.61:45205)
0806 10:54:09.241853 (+ 20043us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:09.241859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:09.242759 (+ 900us) server_negotiation.cc:407] Connection header received
0806 10:54:09.243061 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:09.243067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:09.243159 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:09.243292 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:09.246688 (+ 3396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.247963 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.251275 (+ 3312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.251969 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.252092 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:09.267475 (+ 15383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:09.267483 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:09.267488 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:09.267539 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:09.271228 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:09.272903 (+ 1675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:09.272909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:09.272913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:09.273089 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:09.273699 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:09.273706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:09.273709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:09.274086 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:09.274255 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:09.280585 (+ 6330us) server_negotiation.cc:299] Negotiation successful
0806 10:54:09.280852 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19927,"thread_start_us":19782,"threads_started":1}
I20260806 10:54:09.281543 6713 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:09.281651 6718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:09.221810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:47299 (local address 127.30.218.62:34161)
0806 10:54:09.241822 (+ 20012us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:09.241830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:09.241860 (+ 30us) server_negotiation.cc:407] Connection header received
0806 10:54:09.241944 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:09.241951 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:09.242051 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:09.242207 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:09.244556 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.254735 (+ 10179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.257163 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.258194 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.258346 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:09.263174 (+ 4828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:09.263194 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:09.263199 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:09.263288 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:09.267202 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:09.274624 (+ 7422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:09.274629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:09.274632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:09.274813 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:09.275347 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:09.275352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:09.275356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:09.275774 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:09.280158 (+ 4384us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:09.280343 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 10:54:09.281265 (+ 922us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":19853,"thread_start_us":19707,"threads_started":1}
I20260806 10:54:09.282598 6714 heartbeater.cc:347] Connected to a master server at 127.30.218.62:34161
I20260806 10:54:09.282747 6714 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:09.282892 6713 heartbeater.cc:511] Master 127.30.218.60:34453 requested a full tablet report, sending...
I20260806 10:54:09.283217 6714 heartbeater.cc:511] Master 127.30.218.62:34161 requested a full tablet report, sending...
I20260806 10:54:09.284611 6720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:09.222580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:45205 (local address 127.30.218.3:49203)
0806 10:54:09.242644 (+ 20064us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:09.242667 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:09.242871 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:09.244777 (+ 1906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:09.244781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:09.246144 (+ 1363us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:09.246516 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:09.246531 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.248152 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.248157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:09.251118 (+ 2961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:09.251136 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.252861 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.252869 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:09.253689 (+ 820us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:09.255589 (+ 1900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:09.255625 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:09.267209 (+ 11584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:09.271593 (+ 4384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:09.271598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:09.271601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:09.272746 (+ 1145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:09.273262 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:09.273267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:09.273271 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:09.273434 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:09.274347 (+ 913us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:09.274351 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:09.280249 (+ 5898us) client_negotiation.cc:769] Sending connection context
0806 10:54:09.280420 (+ 171us) client_negotiation.cc:240] Negotiation successful
0806 10:54:09.280765 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19898,"mutex_wait_us":423,"spinlock_wait_cycles":69376,"thread_start_us":19873,"threads_started":1}
I20260806 10:54:09.284777 5430 ts_manager.cc:194] Re-registered known tserver with Master: b9d816dd0f9147479f65e57e4a4f33ba (127.30.218.3:36483)
I20260806 10:54:09.284998 6231 ts_manager.cc:194] Registered new tserver with Master: b9d816dd0f9147479f65e57e4a4f33ba (127.30.218.3:36483)
I20260806 10:54:09.286229 5430 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.218.3@KRBTEST.COM'} at 127.30.218.3:47299
I20260806 10:54:09.286919 6715 heartbeater.cc:347] Connected to a master server at 127.30.218.61:45205
I20260806 10:54:09.287071 6715 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:09.287580 6715 heartbeater.cc:511] Master 127.30.218.61:45205 requested a full tablet report, sending...
I20260806 10:54:09.288954 6037 ts_manager.cc:194] Registered new tserver with Master: b9d816dd0f9147479f65e57e4a4f33ba (127.30.218.3:36483)
I20260806 10:54:09.291185 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:09.245600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35664 (local address 127.30.218.62:34161)
0806 10:54:09.257939 (+ 12339us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:09.257946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:09.257972 (+ 26us) server_negotiation.cc:407] Connection header received
0806 10:54:09.258030 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:09.258036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:09.258130 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:09.258242 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:09.264905 (+ 6663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.266341 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.269918 (+ 3577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.270590 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.270745 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:09.274693 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:09.274700 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:09.274705 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:09.274742 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:09.278393 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:09.279340 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:09.279346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:09.279350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:09.279580 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:09.280141 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:09.280147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:09.280149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:09.288704 (+ 8555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:09.288980 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:09.290478 (+ 1498us) server_negotiation.cc:299] Negotiation successful
0806 10:54:09.290768 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3165,"server-negotiator.queue_time_us":12165,"spinlock_wait_cycles":2944,"thread_start_us":12312,"threads_started":1}
I20260806 10:54:09.295821 6729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:09.245769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.30.218.61:45205)
0806 10:54:09.260380 (+ 14611us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:09.260387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:09.269251 (+ 8864us) server_negotiation.cc:407] Connection header received
0806 10:54:09.269333 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:09.269339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:09.269482 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:09.269598 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:09.276951 (+ 7353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.278249 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.282927 (+ 4678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.283711 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.283850 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:09.288038 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:09.288045 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:09.288050 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:09.288098 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:09.292201 (+ 4103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:09.293159 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:09.293164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:09.293168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:09.293337 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:09.294002 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:09.294008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:09.294011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:09.294400 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:09.294632 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:09.295206 (+ 574us) server_negotiation.cc:299] Negotiation successful
0806 10:54:09.295439 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14458,"thread_start_us":14308,"threads_started":1}
I20260806 10:54:09.301638 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:09.262339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52000 (local address 127.30.218.60:34453)
0806 10:54:09.276431 (+ 14092us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:09.276438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:09.278386 (+ 1948us) server_negotiation.cc:407] Connection header received
0806 10:54:09.278464 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:09.278470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:09.278629 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:09.278729 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:09.285966 (+ 7237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.287128 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.289014 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.289727 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.289835 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:09.294594 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:09.294602 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:09.294606 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:09.294654 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:09.298504 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:09.299365 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:09.299372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:09.299376 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:09.299578 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:09.299965 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:09.299971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:09.299974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:09.300366 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:09.300647 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:09.300961 (+ 314us) server_negotiation.cc:299] Negotiation successful
0806 10:54:09.301233 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13970,"thread_start_us":17509,"threads_started":1}
I20260806 10:54:09.296003 5430 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:54:09.305164 5430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:54:09.341926 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:09.326523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:34939 (local address 127.0.0.1:58726)
0806 10:54:09.329888 (+ 3365us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:09.329904 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:09.330039 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:09.332267 (+ 2228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:09.332273 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:09.332296 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:09.333003 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:09.333013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.335198 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.335205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:09.338567 (+ 3362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:09.338589 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.340619 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.340623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:09.340772 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:09.340795 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 10:54:09.340870 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 10:54:09.341014 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3263,"mutex_wait_us":1,"thread_start_us":3765,"threads_started":1}
I20260806 10:54:09.342391 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:09.326993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58726 (local address 127.30.218.1:34939)
0806 10:54:09.327795 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:09.328696 (+ 901us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:09.330205 (+ 1509us) server_negotiation.cc:407] Connection header received
0806 10:54:09.330345 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:09.330353 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:09.330661 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:09.330893 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:09.333204 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.334974 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.338823 (+ 3849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.340208 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.340305 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:09.340445 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:09.341024 (+ 579us) server_negotiation.cc:299] Negotiation successful
0806 10:54:09.341206 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":475,"threads_started":1}
I20260806 10:54:09.341209 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:09.328542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:42745 (local address 127.0.0.1:35802)
0806 10:54:09.330539 (+ 1997us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:09.330555 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:09.330666 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:09.331640 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:09.331647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:09.331680 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:09.332136 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:09.332150 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.334407 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.334414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:09.337053 (+ 2639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:09.337083 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.340017 (+ 2934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.340025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:09.340248 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:09.340293 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 10:54:09.340389 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 10:54:09.340570 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1886,"mutex_wait_us":25,"thread_start_us":1086,"threads_started":1}
I20260806 10:54:09.341980 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:09.328832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35802 (local address 127.30.218.2:42745)
0806 10:54:09.329344 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:09.329352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:09.330792 (+ 1440us) server_negotiation.cc:407] Connection header received
0806 10:54:09.330914 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:09.330921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:09.331430 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:09.331645 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:09.332379 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.334180 (+ 1801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.337295 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.339362 (+ 2067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.339512 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:09.339760 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:09.341341 (+ 1581us) server_negotiation.cc:299] Negotiation successful
0806 10:54:09.341578 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":254,"threads_started":1}
I20260806 10:54:09.347113 6349 tablet_service.cc:1467] Processing CreateTablet for tablet 30f2a65c31394c6ebf97d56be917529a (DEFAULT_TABLE table=test-table [id=b81b0e20e5dc4e298920555e85a2ec97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:54:09.348850 6349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f2a65c31394c6ebf97d56be917529a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:09.349538 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:09.330040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:36483 (local address 127.0.0.1:53664)
0806 10:54:09.342199 (+ 12159us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:09.342219 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:09.342573 (+ 354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:09.343352 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:09.343357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:09.343384 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:09.343552 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:09.343563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.344964 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.344969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:09.346409 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:09.346422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.348639 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.348646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:09.348856 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:09.348891 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 10:54:09.348991 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 10:54:09.349151 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12051,"thread_start_us":15722,"threads_started":1}
I20260806 10:54:09.349658 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:09.331381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53664 (local address 127.30.218.3:36483)
0806 10:54:09.339748 (+ 8367us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:09.339757 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:09.342315 (+ 2558us) server_negotiation.cc:407] Connection header received
0806 10:54:09.342846 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:09.342854 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:09.343152 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:09.343354 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:09.343687 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.344811 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.346578 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.348179 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.348315 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:09.348534 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:09.349076 (+ 542us) server_negotiation.cc:299] Negotiation successful
0806 10:54:09.349278 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8150,"thread_start_us":7938,"threads_started":1}
I20260806 10:54:09.350265 6503 tablet_service.cc:1467] Processing CreateTablet for tablet 30f2a65c31394c6ebf97d56be917529a (DEFAULT_TABLE table=test-table [id=b81b0e20e5dc4e298920555e85a2ec97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:54:09.351706 6503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f2a65c31394c6ebf97d56be917529a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:09.353736 6647 tablet_service.cc:1467] Processing CreateTablet for tablet 30f2a65c31394c6ebf97d56be917529a (DEFAULT_TABLE table=test-table [id=b81b0e20e5dc4e298920555e85a2ec97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:54:09.355345 6647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f2a65c31394c6ebf97d56be917529a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:09.362888 6746 tablet_bootstrap.cc:492] T 30f2a65c31394c6ebf97d56be917529a P 3478ba34809347f3950b6b5326dd157d: Bootstrap starting.
I20260806 10:54:09.367483 6746 tablet_bootstrap.cc:654] T 30f2a65c31394c6ebf97d56be917529a P 3478ba34809347f3950b6b5326dd157d: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:09.368620 6746 log.cc:824] T 30f2a65c31394c6ebf97d56be917529a P 3478ba34809347f3950b6b5326dd157d: Log is configured to *not* fsync() on all Append() calls
I20260806 10:54:09.371809 6746 tablet_bootstrap.cc:492] T 30f2a65c31394c6ebf97d56be917529a P 3478ba34809347f3950b6b5326dd157d: No bootstrap required, opened a new log
I20260806 10:54:09.371968 6749 tablet_bootstrap.cc:492] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba: Bootstrap starting.
I20260806 10:54:09.372123 6746 ts_tablet_manager.cc:1397] T 30f2a65c31394c6ebf97d56be917529a P 3478ba34809347f3950b6b5326dd157d: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 10:54:09.372048 6747 tablet_bootstrap.cc:492] T 30f2a65c31394c6ebf97d56be917529a P ef6b396ce9db43c49069415d3b6881df: Bootstrap starting.
I20260806 10:54:09.375277 6749 tablet_bootstrap.cc:654] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:09.375985 6747 tablet_bootstrap.cc:654] T 30f2a65c31394c6ebf97d56be917529a P ef6b396ce9db43c49069415d3b6881df: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:09.376703 6749 log.cc:824] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba: Log is configured to *not* fsync() on all Append() calls
I20260806 10:54:09.377259 6747 log.cc:824] T 30f2a65c31394c6ebf97d56be917529a P ef6b396ce9db43c49069415d3b6881df: Log is configured to *not* fsync() on all Append() calls
I20260806 10:54:09.380174 6747 tablet_bootstrap.cc:492] T 30f2a65c31394c6ebf97d56be917529a P ef6b396ce9db43c49069415d3b6881df: No bootstrap required, opened a new log
I20260806 10:54:09.380295 6749 tablet_bootstrap.cc:492] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba: No bootstrap required, opened a new log
I20260806 10:54:09.380491 6747 ts_tablet_manager.cc:1397] T 30f2a65c31394c6ebf97d56be917529a P ef6b396ce9db43c49069415d3b6881df: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260806 10:54:09.380594 6749 ts_tablet_manager.cc:1397] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260806 10:54:09.385070 6746 raft_consensus.cc:348] T 30f2a65c31394c6ebf97d56be917529a P 3478ba34809347f3950b6b5326dd157d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3478ba34809347f3950b6b5326dd157d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34939 } } peers { permanent_uuid: "ef6b396ce9db43c49069415d3b6881df" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42745 } } peers { permanent_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 36483 } }
I20260806 10:54:09.385658 6746 raft_consensus.cc:374] T 30f2a65c31394c6ebf97d56be917529a P 3478ba34809347f3950b6b5326dd157d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:09.385780 6746 raft_consensus.cc:729] T 30f2a65c31394c6ebf97d56be917529a P 3478ba34809347f3950b6b5326dd157d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3478ba34809347f3950b6b5326dd157d, State: Initialized, Role: FOLLOWER
I20260806 10:54:09.386572 6746 consensus_queue.cc:260] T 30f2a65c31394c6ebf97d56be917529a P 3478ba34809347f3950b6b5326dd157d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3478ba34809347f3950b6b5326dd157d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34939 } } peers { permanent_uuid: "ef6b396ce9db43c49069415d3b6881df" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42745 } } peers { permanent_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 36483 } }
I20260806 10:54:09.388641 6746 ts_tablet_manager.cc:1428] T 30f2a65c31394c6ebf97d56be917529a P 3478ba34809347f3950b6b5326dd157d: Time spent starting tablet: real 0.016s user 0.008s sys 0.005s
I20260806 10:54:09.389372 6747 raft_consensus.cc:348] T 30f2a65c31394c6ebf97d56be917529a P ef6b396ce9db43c49069415d3b6881df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3478ba34809347f3950b6b5326dd157d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34939 } } peers { permanent_uuid: "ef6b396ce9db43c49069415d3b6881df" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42745 } } peers { permanent_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 36483 } }
I20260806 10:54:09.389945 6747 raft_consensus.cc:374] T 30f2a65c31394c6ebf97d56be917529a P ef6b396ce9db43c49069415d3b6881df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:09.390056 6747 raft_consensus.cc:729] T 30f2a65c31394c6ebf97d56be917529a P ef6b396ce9db43c49069415d3b6881df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef6b396ce9db43c49069415d3b6881df, State: Initialized, Role: FOLLOWER
I20260806 10:54:09.390827 6747 consensus_queue.cc:260] T 30f2a65c31394c6ebf97d56be917529a P ef6b396ce9db43c49069415d3b6881df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3478ba34809347f3950b6b5326dd157d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34939 } } peers { permanent_uuid: "ef6b396ce9db43c49069415d3b6881df" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42745 } } peers { permanent_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 36483 } }
I20260806 10:54:09.391800 6749 raft_consensus.cc:348] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3478ba34809347f3950b6b5326dd157d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34939 } } peers { permanent_uuid: "ef6b396ce9db43c49069415d3b6881df" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42745 } } peers { permanent_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 36483 } }
I20260806 10:54:09.392290 6749 raft_consensus.cc:374] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:09.392387 6749 raft_consensus.cc:729] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d816dd0f9147479f65e57e4a4f33ba, State: Initialized, Role: FOLLOWER
I20260806 10:54:09.393149 6749 consensus_queue.cc:260] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3478ba34809347f3950b6b5326dd157d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34939 } } peers { permanent_uuid: "ef6b396ce9db43c49069415d3b6881df" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42745 } } peers { permanent_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 36483 } }
I20260806 10:54:09.397228 6714 heartbeater.cc:503] Master 127.30.218.62:34161 was elected leader, sending a full tablet report...
I20260806 10:54:09.398135 6749 ts_tablet_manager.cc:1428] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba: Time spent starting tablet: real 0.017s user 0.017s sys 0.002s
I20260806 10:54:09.400044 6570 heartbeater.cc:503] Master 127.30.218.62:34161 was elected leader, sending a full tablet report...
I20260806 10:54:09.400580 6747 ts_tablet_manager.cc:1428] T 30f2a65c31394c6ebf97d56be917529a P ef6b396ce9db43c49069415d3b6881df: Time spent starting tablet: real 0.020s user 0.012s sys 0.007s
W20260806 10:54:09.459069 6425 tablet.cc:2369] T 30f2a65c31394c6ebf97d56be917529a P 3478ba34809347f3950b6b5326dd157d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:54:09.462292 6754 raft_consensus.cc:482] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:54:09.462599 6754 raft_consensus.cc:504] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3478ba34809347f3950b6b5326dd157d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34939 } } peers { permanent_uuid: "ef6b396ce9db43c49069415d3b6881df" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42745 } } peers { permanent_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 36483 } }
I20260806 10:54:09.464342 6754 leader_election.cc:288] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3478ba34809347f3950b6b5326dd157d (127.30.218.1:34939), ef6b396ce9db43c49069415d3b6881df (127.30.218.2:42745)
I20260806 10:54:09.470022 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:09.464807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:45191 (local address 127.30.218.1:34939)
0806 10:54:09.465054 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:09.465060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:09.465082 (+ 22us) server_negotiation.cc:407] Connection header received
0806 10:54:09.465281 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:09.465284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:09.465363 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:09.465495 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:09.465839 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.466573 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.468310 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.469076 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.469221 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:09.469297 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:09.469629 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 10:54:09.469762 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260806 10:54:09.470073 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:09.464881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:60931 (local address 127.30.218.2:42745)
0806 10:54:09.465067 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:09.465072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:09.465093 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:54:09.465167 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:09.465173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:09.465248 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:09.465318 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:09.465838 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.466559 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.468323 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.469112 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.469262 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:09.469333 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:09.469717 (+ 384us) server_negotiation.cc:299] Negotiation successful
0806 10:54:09.469849 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 10:54:09.470052 6723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:09.464592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:34939 (local address 127.30.218.3:45191)
0806 10:54:09.464998 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:09.465015 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:09.465123 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:09.465515 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:09.465519 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:09.465539 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:09.465711 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:09.465718 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.466715 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.466718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:09.468153 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:09.468166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.469259 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.469265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:09.469483 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:09.469511 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 10:54:09.469583 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 10:54:09.469708 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":209}
I20260806 10:54:09.470072 6720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:09.464592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:42745 (local address 127.30.218.3:60931)
0806 10:54:09.464861 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:09.464882 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:09.465021 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:09.465371 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:09.465376 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:09.465402 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:09.465688 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:09.465699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.466697 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.466707 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:09.468164 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:09.468177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.469270 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.469276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:09.469545 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:09.469573 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 10:54:09.469674 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 10:54:09.469806 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":117}
I20260806 10:54:09.471153 6524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f2a65c31394c6ebf97d56be917529a" candidate_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef6b396ce9db43c49069415d3b6881df" is_pre_election: true
I20260806 10:54:09.471208 6370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f2a65c31394c6ebf97d56be917529a" candidate_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3478ba34809347f3950b6b5326dd157d" is_pre_election: true
W20260806 10:54:09.471622 6716 tablet.cc:2369] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:54:09.471673 6524 raft_consensus.cc:2454] T 30f2a65c31394c6ebf97d56be917529a P ef6b396ce9db43c49069415d3b6881df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d816dd0f9147479f65e57e4a4f33ba in term 0.
I20260806 10:54:09.471704 6370 raft_consensus.cc:2454] T 30f2a65c31394c6ebf97d56be917529a P 3478ba34809347f3950b6b5326dd157d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d816dd0f9147479f65e57e4a4f33ba in term 0.
I20260806 10:54:09.472409 6599 leader_election.cc:302] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3478ba34809347f3950b6b5326dd157d, b9d816dd0f9147479f65e57e4a4f33ba; no voters:
I20260806 10:54:09.472818 6754 raft_consensus.cc:2781] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:54:09.472966 6754 raft_consensus.cc:482] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:54:09.473073 6754 raft_consensus.cc:3037] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:09.475286 6754 raft_consensus.cc:504] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3478ba34809347f3950b6b5326dd157d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34939 } } peers { permanent_uuid: "ef6b396ce9db43c49069415d3b6881df" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42745 } } peers { permanent_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 36483 } }
I20260806 10:54:09.475873 6754 leader_election.cc:288] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba [CANDIDATE]: Term 1 election: Requested vote from peers 3478ba34809347f3950b6b5326dd157d (127.30.218.1:34939), ef6b396ce9db43c49069415d3b6881df (127.30.218.2:42745)
I20260806 10:54:09.476397 6370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f2a65c31394c6ebf97d56be917529a" candidate_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3478ba34809347f3950b6b5326dd157d"
I20260806 10:54:09.476405 6524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f2a65c31394c6ebf97d56be917529a" candidate_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef6b396ce9db43c49069415d3b6881df"
I20260806 10:54:09.476591 6370 raft_consensus.cc:3037] T 30f2a65c31394c6ebf97d56be917529a P 3478ba34809347f3950b6b5326dd157d [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:09.476585 6524 raft_consensus.cc:3037] T 30f2a65c31394c6ebf97d56be917529a P ef6b396ce9db43c49069415d3b6881df [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:09.478916 6370 raft_consensus.cc:2454] T 30f2a65c31394c6ebf97d56be917529a P 3478ba34809347f3950b6b5326dd157d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d816dd0f9147479f65e57e4a4f33ba in term 1.
I20260806 10:54:09.478972 6524 raft_consensus.cc:2454] T 30f2a65c31394c6ebf97d56be917529a P ef6b396ce9db43c49069415d3b6881df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d816dd0f9147479f65e57e4a4f33ba in term 1.
I20260806 10:54:09.479369 6599 leader_election.cc:302] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3478ba34809347f3950b6b5326dd157d, b9d816dd0f9147479f65e57e4a4f33ba, ef6b396ce9db43c49069415d3b6881df; no voters:
I20260806 10:54:09.479698 6754 raft_consensus.cc:2781] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:54:09.480443 6754 raft_consensus.cc:686] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba [term 1 LEADER]: Becoming Leader. State: Replica: b9d816dd0f9147479f65e57e4a4f33ba, State: Running, Role: LEADER
I20260806 10:54:09.481048 6754 consensus_queue.cc:237] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3478ba34809347f3950b6b5326dd157d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34939 } } peers { permanent_uuid: "ef6b396ce9db43c49069415d3b6881df" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42745 } } peers { permanent_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 36483 } }
I20260806 10:54:09.485802 5430 catalog_manager.cc:5495] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba reported cstate change: term changed from 0 to 1, leader changed from <none> to b9d816dd0f9147479f65e57e4a4f33ba (127.30.218.3). New cstate: current_term: 1 leader_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3478ba34809347f3950b6b5326dd157d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6b396ce9db43c49069415d3b6881df" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 36483 } health_report { overall_health: HEALTHY } } }
I20260806 10:54:09.531116 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:09.525787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.30.218.3:36483)
0806 10:54:09.525986 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:09.525990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:09.526009 (+ 19us) server_negotiation.cc:407] Connection header received
0806 10:54:09.526280 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:09.526284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:09.526377 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:09.526516 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:54:09.526923 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.527641 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.529026 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.529511 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.529612 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:09.530096 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:54:09.530637 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:54:09.530758 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:09.530816 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 10:54:09.530894 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 10:54:09.540217 6370 raft_consensus.cc:1261] T 30f2a65c31394c6ebf97d56be917529a P 3478ba34809347f3950b6b5326dd157d [term 1 FOLLOWER]: Refusing update from remote peer b9d816dd0f9147479f65e57e4a4f33ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:54:09.540215 6524 raft_consensus.cc:1261] T 30f2a65c31394c6ebf97d56be917529a P ef6b396ce9db43c49069415d3b6881df [term 1 FOLLOWER]: Refusing update from remote peer b9d816dd0f9147479f65e57e4a4f33ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:54:09.541342 6755 consensus_queue.cc:1035] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6b396ce9db43c49069415d3b6881df" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:54:09.541769 6754 consensus_queue.cc:1035] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "3478ba34809347f3950b6b5326dd157d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:54:09.557403 6759 mvcc.cc:205] Tried to move back new op lower bound from 7315511908505079808 to 7315511908278779904. Current Snapshot: MvccSnapshot[applied={T|T < 7315511908505079808}]
I20260806 10:54:09.566515 6761 mvcc.cc:205] Tried to move back new op lower bound from 7315511908505079808 to 7315511908278779904. Current Snapshot: MvccSnapshot[applied={T|T < 7315511908505079808 or (T in {7315511908505079808})}]
I20260806 10:54:09.574441 5430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:54:09.574708 5430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:54:09.583434 5430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b824ac0a7c94632b4d0aea8540f15bd: Sending DeleteTablet for 3 replicas of tablet 30f2a65c31394c6ebf97d56be917529a
I20260806 10:54:09.584517 6349 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f2a65c31394c6ebf97d56be917529a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:54:09 UTC) from {username='oryx', principal='oryx/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:58726
I20260806 10:54:09.584546 6503 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f2a65c31394c6ebf97d56be917529a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:54:09 UTC) from {username='oryx', principal='oryx/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:35802
I20260806 10:54:09.585067 6647 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f2a65c31394c6ebf97d56be917529a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:54:09 UTC) from {username='oryx', principal='oryx/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:53664
I20260806 10:54:09.600010 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 5397
I20260806 10:54:09.601050 6771 tablet_replica.cc:330] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba: stopping tablet replica
I20260806 10:54:09.601327 5460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:09.601634 6771 raft_consensus.cc:2229] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:54:09.602466 6771 raft_consensus.cc:2258] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:54:09.607144 6769 tablet_replica.cc:330] T 30f2a65c31394c6ebf97d56be917529a P 3478ba34809347f3950b6b5326dd157d: stopping tablet replica
I20260806 10:54:09.607712 6769 raft_consensus.cc:2229] T 30f2a65c31394c6ebf97d56be917529a P 3478ba34809347f3950b6b5326dd157d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:54:09.608333 6769 raft_consensus.cc:2258] T 30f2a65c31394c6ebf97d56be917529a P 3478ba34809347f3950b6b5326dd157d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:54:09.612038 6770 tablet_replica.cc:330] T 30f2a65c31394c6ebf97d56be917529a P ef6b396ce9db43c49069415d3b6881df: stopping tablet replica
I20260806 10:54:09.612447 6770 raft_consensus.cc:2229] T 30f2a65c31394c6ebf97d56be917529a P ef6b396ce9db43c49069415d3b6881df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:54:09.612923 6770 raft_consensus.cc:2258] T 30f2a65c31394c6ebf97d56be917529a P ef6b396ce9db43c49069415d3b6881df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:54:09.612946 6771 ts_tablet_manager.cc:1905] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:54:09.615219 6770 ts_tablet_manager.cc:1905] T 30f2a65c31394c6ebf97d56be917529a P ef6b396ce9db43c49069415d3b6881df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:54:09.616109 6769 ts_tablet_manager.cc:1905] T 30f2a65c31394c6ebf97d56be917529a P 3478ba34809347f3950b6b5326dd157d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:54:09.622864 6770 ts_tablet_manager.cc:1918] T 30f2a65c31394c6ebf97d56be917529a P ef6b396ce9db43c49069415d3b6881df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:54:09.623080 6770 log.cc:1196] T 30f2a65c31394c6ebf97d56be917529a P ef6b396ce9db43c49069415d3b6881df: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/30f2a65c31394c6ebf97d56be917529a
I20260806 10:54:09.623740 6770 ts_tablet_manager.cc:1939] T 30f2a65c31394c6ebf97d56be917529a P ef6b396ce9db43c49069415d3b6881df: Deleting consensus metadata
I20260806 10:54:09.623863 6771 ts_tablet_manager.cc:1918] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:54:09.624063 6771 log.cc:1196] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/30f2a65c31394c6ebf97d56be917529a
I20260806 10:54:09.624703 6771 ts_tablet_manager.cc:1939] T 30f2a65c31394c6ebf97d56be917529a P b9d816dd0f9147479f65e57e4a4f33ba: Deleting consensus metadata
I20260806 10:54:09.633104 6769 ts_tablet_manager.cc:1918] T 30f2a65c31394c6ebf97d56be917529a P 3478ba34809347f3950b6b5326dd157d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:54:09.633332 6769 log.cc:1196] T 30f2a65c31394c6ebf97d56be917529a P 3478ba34809347f3950b6b5326dd157d: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/wal/wals/30f2a65c31394c6ebf97d56be917529a
I20260806 10:54:09.634006 6769 ts_tablet_manager.cc:1939] T 30f2a65c31394c6ebf97d56be917529a P 3478ba34809347f3950b6b5326dd157d: Deleting consensus metadata
I20260806 10:54:09.636631 6729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:09.591843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37318 (local address 127.30.218.61:45205)
0806 10:54:09.592067 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:09.592073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:09.598311 (+ 6238us) server_negotiation.cc:407] Connection header received
0806 10:54:09.598759 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:09.598768 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:09.598904 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:09.598987 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:09.601664 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.602702 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.608492 (+ 5790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.609154 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.609274 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:09.625787 (+ 16513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:09.625797 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:09.625803 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:09.625859 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:09.629604 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:09.630705 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:09.630712 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:09.630715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:09.630908 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:09.631384 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:09.631389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:09.631393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:09.631807 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:09.632061 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:09.636016 (+ 3955us) server_negotiation.cc:299] Negotiation successful
0806 10:54:09.636294 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 10:54:09.652550 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:09.598527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52002 (local address 127.30.218.60:34453)
0806 10:54:09.598749 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:09.598755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:09.633404 (+ 34649us) server_negotiation.cc:407] Connection header received
0806 10:54:09.633735 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:09.633742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:09.633842 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:09.633909 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:09.636924 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.638155 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.640475 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.641203 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.641340 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:09.645145 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:09.645154 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:09.645159 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:09.645211 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:09.649014 (+ 3803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:09.650077 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:09.650083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:09.650087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:09.650292 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:09.650775 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:09.650781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:09.650785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:09.651187 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:09.651455 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:09.651886 (+ 431us) server_negotiation.cc:299] Negotiation successful
0806 10:54:09.652158 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 10:54:09.758255 5414 catalog_manager.cc:4841] TS 3478ba34809347f3950b6b5326dd157d (127.30.218.1:34939): tablet 30f2a65c31394c6ebf97d56be917529a (table test-table [id=b81b0e20e5dc4e298920555e85a2ec97]) successfully deleted
I20260806 10:54:09.758275 5417 catalog_manager.cc:4841] TS b9d816dd0f9147479f65e57e4a4f33ba (127.30.218.3:36483): tablet 30f2a65c31394c6ebf97d56be917529a (table test-table [id=b81b0e20e5dc4e298920555e85a2ec97]) successfully deleted
I20260806 10:54:09.758781 5417 catalog_manager.cc:4841] TS ef6b396ce9db43c49069415d3b6881df (127.30.218.2:42745): tablet 30f2a65c31394c6ebf97d56be917529a (table test-table [id=b81b0e20e5dc4e298920555e85a2ec97]) successfully deleted
I20260806 10:54:09.758957 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 5397
I20260806 10:54:09.805775 6729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:09.787436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37320 (local address 127.30.218.61:45205)
0806 10:54:09.787638 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:09.787645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:09.788448 (+ 803us) server_negotiation.cc:407] Connection header received
0806 10:54:09.789823 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:09.789830 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:09.789922 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:09.789996 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:09.792294 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.793225 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.794634 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.795313 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.795454 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:09.799440 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:09.799448 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:09.799453 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:09.799502 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:09.802788 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:09.803584 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:09.803590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:09.803593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:09.803740 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:09.804126 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:09.804132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:09.804135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:09.804462 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:09.804694 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:09.805085 (+ 391us) server_negotiation.cc:299] Negotiation successful
0806 10:54:09.805343 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 10:54:09.808290 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:09.788718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52018 (local address 127.30.218.60:34453)
0806 10:54:09.788939 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:09.788944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:09.789574 (+ 630us) server_negotiation.cc:407] Connection header received
0806 10:54:09.792472 (+ 2898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:09.792478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:09.792571 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:09.792683 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:09.794835 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.795871 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.797263 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:09.797868 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:09.797989 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:09.801187 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:09.801195 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:09.801200 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:09.801248 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:09.804688 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:09.805810 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:09.805816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:09.805820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:09.806008 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:09.806543 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:09.806549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:09.806552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:09.807165 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:09.807448 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:09.807712 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 10:54:09.807956 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 10:54:10.563468 6781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:10.562464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.30.218.2:57147)
0806 10:54:10.563069 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:10.563215 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":342,"threads_started":1}
W20260806 10:54:10.564379 6570 heartbeater.cc:644] Failed to heartbeat to 127.30.218.62:34161 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
I20260806 10:54:10.567623 6782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:10.566818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.30.218.1:53549)
0806 10:54:10.567298 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:10.567440 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":241,"threads_started":1}
W20260806 10:54:10.568058 6417 heartbeater.cc:644] Failed to heartbeat to 127.30.218.62:34161 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
I20260806 10:54:10.571009 6784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:10.569958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.30.218.3:44697)
0806 10:54:10.570490 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:10.570681 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":262,"threads_started":1}
W20260806 10:54:10.571516 6714 heartbeater.cc:644] Failed to heartbeat to 127.30.218.62:34161 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
I20260806 10:54:11.081782 6785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b824ac0a7c94632b4d0aea8540f15bd)
I20260806 10:54:11.082067 6785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: false } }
I20260806 10:54:11.083146 6785 leader_election.cc:288] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b824ac0a7c94632b4d0aea8540f15bd (127.30.218.62:34161), f5fa1e7cf26345d2b7538c6d34ea8149 (127.30.218.61:45205)
I20260806 10:54:11.095605 6786 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b824ac0a7c94632b4d0aea8540f15bd)
I20260806 10:54:11.096276 6788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.083389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.0.0.1:36050)
0806 10:54:11.095477 (+ 12088us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.095862 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11950,"thread_start_us":12154,"threads_started":1}
I20260806 10:54:11.096148 6786 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: false } }
W20260806 10:54:11.096853 6215 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
I20260806 10:54:11.100106 6786 leader_election.cc:288] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b824ac0a7c94632b4d0aea8540f15bd (127.30.218.62:34161), 7b65bc5a92ba4967bf17cef40391bb14 (127.30.218.60:34453)
I20260806 10:54:11.101352 6791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.099091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.0.0.1:36082)
0806 10:54:11.100874 (+ 1783us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.101039 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1657,"thread_start_us":410,"threads_started":1}
I20260806 10:54:11.101519 6788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.101016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.0.0.1:36084)
0806 10:54:11.101204 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.101318 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260806 10:54:11.101768 6215 leader_election.cc:334] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4b824ac0a7c94632b4d0aea8540f15bd (127.30.218.62:34161): Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
W20260806 10:54:11.101917 6020 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
I20260806 10:54:11.104105 6791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.103655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.0.0.1:36090)
0806 10:54:11.103847 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.103942 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 10:54:11.104276 6020 leader_election.cc:334] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4b824ac0a7c94632b4d0aea8540f15bd (127.30.218.62:34161): Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
I20260806 10:54:11.106446 6787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.083556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:45205 (local address 127.0.0.1:37332)
0806 10:54:11.095896 (+ 12340us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.095915 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:11.096162 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:11.096891 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:11.096896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:11.096959 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:11.097362 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:11.097379 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.101685 (+ 4306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.101692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:11.103974 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:11.103991 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.105788 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.105795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:11.105935 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:11.105963 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 10:54:11.106062 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 10:54:11.106176 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12169,"thread_start_us":11591,"threads_started":1}
I20260806 10:54:11.106702 6789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.083673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37332 (local address 127.30.218.61:45205)
0806 10:54:11.094818 (+ 11145us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:11.094829 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:11.096067 (+ 1238us) server_negotiation.cc:407] Connection header received
0806 10:54:11.096413 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:11.096419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:11.096525 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:11.096674 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:11.097641 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.099093 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.104152 (+ 5059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.105633 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.105761 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:11.105869 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:11.106118 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 10:54:11.106264 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":10925,"thread_start_us":11129,"threads_started":1}
I20260806 10:54:11.107401 6047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7b65bc5a92ba4967bf17cef40391bb14" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" is_pre_election: true
I20260806 10:54:11.107820 6047 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b65bc5a92ba4967bf17cef40391bb14 in term 1.
I20260806 10:54:11.108435 6218 leader_election.cc:302] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b65bc5a92ba4967bf17cef40391bb14, f5fa1e7cf26345d2b7538c6d34ea8149; no voters: 4b824ac0a7c94632b4d0aea8540f15bd
I20260806 10:54:11.108832 6785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 10:54:11.109020 6785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4b824ac0a7c94632b4d0aea8540f15bd)
I20260806 10:54:11.109081 6792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.098986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:34453 (local address 127.0.0.1:52032)
0806 10:54:11.099745 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.099765 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:11.099905 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:11.102449 (+ 2544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:11.102455 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:11.102494 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:11.102985 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:11.103003 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.104506 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.104511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:11.106637 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:11.106653 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.108269 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.108276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:11.108454 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:11.108483 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 10:54:11.108565 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 10:54:11.108717 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":1629,"threads_started":1}
I20260806 10:54:11.109247 6793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.100260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52032 (local address 127.30.218.60:34453)
0806 10:54:11.102070 (+ 1810us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:11.102076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:11.102105 (+ 29us) server_negotiation.cc:407] Connection header received
0806 10:54:11.102181 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:11.102186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:11.102282 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:11.102393 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:11.103198 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.104344 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.106836 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.108050 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.108182 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:11.108303 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:11.108614 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 10:54:11.108762 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1673,"thread_start_us":1108,"threads_started":1}
I20260806 10:54:11.109242 6785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [term 1 FOLLOWER]: Advancing to term 2
I20260806 10:54:11.109967 6241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7b65bc5a92ba4967bf17cef40391bb14" is_pre_election: true
I20260806 10:54:11.112223 6785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: false } }
I20260806 10:54:11.112717 6241 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f5fa1e7cf26345d2b7538c6d34ea8149 in current term 2: Already voted for candidate 7b65bc5a92ba4967bf17cef40391bb14 in this term.
I20260806 10:54:11.112977 6785 leader_election.cc:288] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [CANDIDATE]: Term 2 election: Requested vote from peers 4b824ac0a7c94632b4d0aea8540f15bd (127.30.218.62:34161), f5fa1e7cf26345d2b7538c6d34ea8149 (127.30.218.61:45205)
I20260806 10:54:11.113626 6787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.113137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.0.0.1:36094)
0806 10:54:11.113300 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.113432 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 10:54:11.113636 6047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7b65bc5a92ba4967bf17cef40391bb14" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149"
I20260806 10:54:11.113914 6047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [term 1 FOLLOWER]: Advancing to term 2
I20260806 10:54:11.114153 6023 leader_election.cc:302] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f5fa1e7cf26345d2b7538c6d34ea8149; no voters: 4b824ac0a7c94632b4d0aea8540f15bd, 7b65bc5a92ba4967bf17cef40391bb14
I20260806 10:54:11.116479 6047 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b65bc5a92ba4967bf17cef40391bb14 in term 2.
I20260806 10:54:11.116704 6786 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 10:54:11.116966 6218 leader_election.cc:302] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b65bc5a92ba4967bf17cef40391bb14, f5fa1e7cf26345d2b7538c6d34ea8149; no voters:
I20260806 10:54:11.117313 6785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 10:54:11.117901 6785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [term 2 LEADER]: Becoming Leader. State: Replica: 7b65bc5a92ba4967bf17cef40391bb14, State: Running, Role: LEADER
I20260806 10:54:11.118010 6787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.117488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.0.0.1:36100)
0806 10:54:11.117656 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.117805 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 10:54:11.118156 6215 leader_election.cc:334] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4b824ac0a7c94632b4d0aea8540f15bd (127.30.218.62:34161): Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
I20260806 10:54:11.118453 6785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: false } }
I20260806 10:54:11.120474 6795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b65bc5a92ba4967bf17cef40391bb14. Latest consensus state: current_term: 2 leader_uuid: "7b65bc5a92ba4967bf17cef40391bb14" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: false } } }
I20260806 10:54:11.120886 6795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [sys.catalog]: This master's current role is: LEADER
I20260806 10:54:11.121677 6797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:54:11.126869 6797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:54:11.127978 6797 catalog_manager.cc:1259] Loaded cluster ID: fd65f6dd547845159ea0ec495152a496
I20260806 10:54:11.128072 6797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:54:11.129437 6797 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:54:11.130399 6797 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14: Loaded TSK: 0
I20260806 10:54:11.131595 6797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:54:11.164281 6230 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:54:11.172544 6230 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:54:11.177810 6787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.177307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.0.0.1:36104)
0806 10:54:11.177522 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.177624 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 10:54:11.177954 6047 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [term 2 FOLLOWER]: Refusing update from remote peer 7b65bc5a92ba4967bf17cef40391bb14: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260806 10:54:11.178905 6787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.178483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.0.0.1:36108)
0806 10:54:11.178660 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.178741 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 10:54:11.178864 6795 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260806 10:54:11.179160 6215 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 -> Peer 4b824ac0a7c94632b4d0aea8540f15bd (127.30.218.62:34161): Couldn't send request to peer 4b824ac0a7c94632b4d0aea8540f15bd. Status: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 10:54:11.182178 6786 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b65bc5a92ba4967bf17cef40391bb14. Latest consensus state: current_term: 2 leader_uuid: "7b65bc5a92ba4967bf17cef40391bb14" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: false } } }
I20260806 10:54:11.182454 6786 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:54:11.184051 6795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "7b65bc5a92ba4967bf17cef40391bb14" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: false } } }
I20260806 10:54:11.184404 6795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14 [sys.catalog]: This master's current role is: LEADER
I20260806 10:54:11.186089 6786 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "7b65bc5a92ba4967bf17cef40391bb14" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b824ac0a7c94632b4d0aea8540f15bd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34161 } } peers { permanent_uuid: "f5fa1e7cf26345d2b7538c6d34ea8149" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "7b65bc5a92ba4967bf17cef40391bb14" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 34453 } attrs { promote: false } } }
I20260806 10:54:11.186390 6786 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5fa1e7cf26345d2b7538c6d34ea8149 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:54:11.211941 6805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.194516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.30.218.1:34939)
0806 10:54:11.205893 (+ 11377us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:11.205899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:11.205919 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:54:11.205995 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:11.205999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:11.206088 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:11.206196 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:11.206769 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.207691 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.209925 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.210910 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.211041 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:11.211176 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:11.211464 (+ 288us) server_negotiation.cc:299] Negotiation successful
0806 10:54:11.211620 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11157,"thread_start_us":10870,"threads_started":1}
I20260806 10:54:11.211932 6788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.194108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:34939 (local address 127.0.0.1:58730)
0806 10:54:11.194245 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.194264 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:11.194387 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:11.206248 (+ 11861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:11.206254 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:11.206283 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:11.206601 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:11.206610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.207832 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.207838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:11.209737 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:11.209751 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.211124 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.211130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:11.211294 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:11.211321 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 10:54:11.211416 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 10:54:11.211587 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260806 10:54:11.212963 6349 tablet_service.cc:1467] Processing CreateTablet for tablet 24670bda15d54efaa5bd278f2a094ab1 (DEFAULT_TABLE table=test-table [id=b14cf414e53541bca2bb6cbe258fac96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:54:11.213598 6349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24670bda15d54efaa5bd278f2a094ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:11.216668 6804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.193811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35812 (local address 127.30.218.2:42745)
0806 10:54:11.209553 (+ 15742us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:11.209561 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:11.209586 (+ 25us) server_negotiation.cc:407] Connection header received
0806 10:54:11.209681 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:11.209688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:11.209780 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:11.209912 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:11.210278 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.211609 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.214277 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.215494 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.215668 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:11.215778 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:11.216059 (+ 281us) server_negotiation.cc:299] Negotiation successful
0806 10:54:11.216226 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15540,"thread_start_us":16235,"threads_started":1}
I20260806 10:54:11.217938 6806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.194664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:36483 (local address 127.0.0.1:53690)
0806 10:54:11.204796 (+ 10132us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.204821 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:11.204994 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:11.212154 (+ 7160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:11.212158 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:11.212190 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:11.212618 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:11.212631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.214165 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.214171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:11.215464 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:11.215476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.217113 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.217119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:11.217300 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:11.217335 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 10:54:11.217402 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 10:54:11.217567 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9968,"thread_start_us":10425,"threads_started":1}
I20260806 10:54:11.218971 6807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.194902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53690 (local address 127.30.218.3:36483)
0806 10:54:11.211660 (+ 16758us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:11.211667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:11.211688 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:54:11.211774 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:11.211780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:11.211871 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:11.212785 (+ 914us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:11.213012 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.214045 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.215633 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.216893 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.218300 (+ 1407us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:11.218426 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:11.218483 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 10:54:11.218628 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16605,"thread_start_us":17065,"threads_started":1}
I20260806 10:54:11.219841 6647 tablet_service.cc:1467] Processing CreateTablet for tablet 24670bda15d54efaa5bd278f2a094ab1 (DEFAULT_TABLE table=test-table [id=b14cf414e53541bca2bb6cbe258fac96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:54:11.220386 6647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24670bda15d54efaa5bd278f2a094ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:11.220716 6808 tablet_bootstrap.cc:492] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d: Bootstrap starting.
I20260806 10:54:11.221516 6787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.193652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:42745 (local address 127.0.0.1:35812)
0806 10:54:11.194032 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.194056 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:11.194184 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:11.209925 (+ 15741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:11.209929 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:11.209960 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:11.210130 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:11.210140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.211779 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.211785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:11.213712 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:11.213727 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.215689 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.215695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:11.215883 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:11.215920 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 10:54:11.216020 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 10:54:11.216172 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250}
I20260806 10:54:11.222476 6503 tablet_service.cc:1467] Processing CreateTablet for tablet 24670bda15d54efaa5bd278f2a094ab1 (DEFAULT_TABLE table=test-table [id=b14cf414e53541bca2bb6cbe258fac96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:54:11.223026 6503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24670bda15d54efaa5bd278f2a094ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:11.223208 6808 tablet_bootstrap.cc:654] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:11.225961 6808 tablet_bootstrap.cc:492] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d: No bootstrap required, opened a new log
I20260806 10:54:11.226142 6808 ts_tablet_manager.cc:1397] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 10:54:11.227087 6808 raft_consensus.cc:348] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6b396ce9db43c49069415d3b6881df" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42745 } } peers { permanent_uuid: "3478ba34809347f3950b6b5326dd157d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34939 } } peers { permanent_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 36483 } }
I20260806 10:54:11.227372 6808 raft_consensus.cc:374] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:11.227466 6808 raft_consensus.cc:729] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3478ba34809347f3950b6b5326dd157d, State: Initialized, Role: FOLLOWER
I20260806 10:54:11.227754 6808 consensus_queue.cc:260] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6b396ce9db43c49069415d3b6881df" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42745 } } peers { permanent_uuid: "3478ba34809347f3950b6b5326dd157d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34939 } } peers { permanent_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 36483 } }
I20260806 10:54:11.230389 6810 tablet_bootstrap.cc:492] T 24670bda15d54efaa5bd278f2a094ab1 P b9d816dd0f9147479f65e57e4a4f33ba: Bootstrap starting.
I20260806 10:54:11.233032 6810 tablet_bootstrap.cc:654] T 24670bda15d54efaa5bd278f2a094ab1 P b9d816dd0f9147479f65e57e4a4f33ba: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:11.236773 6808 ts_tablet_manager.cc:1428] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d: Time spent starting tablet: real 0.010s user 0.000s sys 0.004s
I20260806 10:54:11.237075 6416 heartbeater.cc:503] Master 127.30.218.60:34453 was elected leader, sending a full tablet report...
I20260806 10:54:11.238498 6812 tablet_bootstrap.cc:492] T 24670bda15d54efaa5bd278f2a094ab1 P ef6b396ce9db43c49069415d3b6881df: Bootstrap starting.
I20260806 10:54:11.241758 6812 tablet_bootstrap.cc:654] T 24670bda15d54efaa5bd278f2a094ab1 P ef6b396ce9db43c49069415d3b6881df: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:11.242610 6813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.230387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.30.218.1:53999)
0806 10:54:11.242092 (+ 11705us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.242305 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11510,"threads_started":1}
I20260806 10:54:11.243989 6813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.243514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.30.218.1:55663)
0806 10:54:11.243695 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.243795 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 10:54:11.244513 6810 tablet_bootstrap.cc:492] T 24670bda15d54efaa5bd278f2a094ab1 P b9d816dd0f9147479f65e57e4a4f33ba: No bootstrap required, opened a new log
I20260806 10:54:11.244673 6810 ts_tablet_manager.cc:1397] T 24670bda15d54efaa5bd278f2a094ab1 P b9d816dd0f9147479f65e57e4a4f33ba: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260806 10:54:11.245527 6810 raft_consensus.cc:348] T 24670bda15d54efaa5bd278f2a094ab1 P b9d816dd0f9147479f65e57e4a4f33ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6b396ce9db43c49069415d3b6881df" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42745 } } peers { permanent_uuid: "3478ba34809347f3950b6b5326dd157d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34939 } } peers { permanent_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 36483 } }
I20260806 10:54:11.245738 6810 raft_consensus.cc:374] T 24670bda15d54efaa5bd278f2a094ab1 P b9d816dd0f9147479f65e57e4a4f33ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:11.245838 6810 raft_consensus.cc:729] T 24670bda15d54efaa5bd278f2a094ab1 P b9d816dd0f9147479f65e57e4a4f33ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d816dd0f9147479f65e57e4a4f33ba, State: Initialized, Role: FOLLOWER
I20260806 10:54:11.246083 6810 consensus_queue.cc:260] T 24670bda15d54efaa5bd278f2a094ab1 P b9d816dd0f9147479f65e57e4a4f33ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6b396ce9db43c49069415d3b6881df" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42745 } } peers { permanent_uuid: "3478ba34809347f3950b6b5326dd157d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34939 } } peers { permanent_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 36483 } }
I20260806 10:54:11.246351 6812 tablet_bootstrap.cc:492] T 24670bda15d54efaa5bd278f2a094ab1 P ef6b396ce9db43c49069415d3b6881df: No bootstrap required, opened a new log
I20260806 10:54:11.246505 6812 ts_tablet_manager.cc:1397] T 24670bda15d54efaa5bd278f2a094ab1 P ef6b396ce9db43c49069415d3b6881df: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 10:54:11.246946 6810 ts_tablet_manager.cc:1428] T 24670bda15d54efaa5bd278f2a094ab1 P b9d816dd0f9147479f65e57e4a4f33ba: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 10:54:11.247300 6812 raft_consensus.cc:348] T 24670bda15d54efaa5bd278f2a094ab1 P ef6b396ce9db43c49069415d3b6881df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6b396ce9db43c49069415d3b6881df" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42745 } } peers { permanent_uuid: "3478ba34809347f3950b6b5326dd157d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34939 } } peers { permanent_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 36483 } }
I20260806 10:54:11.247568 6812 raft_consensus.cc:374] T 24670bda15d54efaa5bd278f2a094ab1 P ef6b396ce9db43c49069415d3b6881df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:11.247689 6812 raft_consensus.cc:729] T 24670bda15d54efaa5bd278f2a094ab1 P ef6b396ce9db43c49069415d3b6881df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef6b396ce9db43c49069415d3b6881df, State: Initialized, Role: FOLLOWER
I20260806 10:54:11.247934 6812 consensus_queue.cc:260] T 24670bda15d54efaa5bd278f2a094ab1 P ef6b396ce9db43c49069415d3b6881df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6b396ce9db43c49069415d3b6881df" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42745 } } peers { permanent_uuid: "3478ba34809347f3950b6b5326dd157d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34939 } } peers { permanent_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 36483 } }
I20260806 10:54:11.249712 6713 heartbeater.cc:503] Master 127.30.218.60:34453 was elected leader, sending a full tablet report...
I20260806 10:54:11.251164 6812 ts_tablet_manager.cc:1428] T 24670bda15d54efaa5bd278f2a094ab1 P ef6b396ce9db43c49069415d3b6881df: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 10:54:11.254181 6569 heartbeater.cc:503] Master 127.30.218.60:34453 was elected leader, sending a full tablet report...
I20260806 10:54:11.258490 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.247283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.30.218.3:45557)
0806 10:54:11.258019 (+ 10736us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.258208 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10592,"thread_start_us":10457,"threads_started":1}
I20260806 10:54:11.259732 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.259334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.30.218.3:51523)
0806 10:54:11.259514 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.259603 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 10:54:11.262480 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.252305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.30.218.2:36557)
0806 10:54:11.261950 (+ 9645us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.262157 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9519,"thread_start_us":9341,"threads_started":1}
I20260806 10:54:11.263558 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.263131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.30.218.2:47007)
0806 10:54:11.263334 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.263409 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 10:54:11.319540 6811 raft_consensus.cc:482] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:54:11.319887 6811 raft_consensus.cc:504] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6b396ce9db43c49069415d3b6881df" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42745 } } peers { permanent_uuid: "3478ba34809347f3950b6b5326dd157d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34939 } } peers { permanent_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 36483 } }
I20260806 10:54:11.322036 6811 leader_election.cc:288] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef6b396ce9db43c49069415d3b6881df (127.30.218.2:42745), b9d816dd0f9147479f65e57e4a4f33ba (127.30.218.3:36483)
W20260806 10:54:11.329173 6572 tablet.cc:2369] T 24670bda15d54efaa5bd278f2a094ab1 P ef6b396ce9db43c49069415d3b6881df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:54:11.329548 6804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.322522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:49981 (local address 127.30.218.2:42745)
0806 10:54:11.322735 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:11.322742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:11.322805 (+ 63us) server_negotiation.cc:407] Connection header received
0806 10:54:11.323153 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:11.323159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:11.323270 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:11.323378 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:11.324236 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.325180 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.327278 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.328397 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.328553 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:11.328623 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:11.329018 (+ 395us) server_negotiation.cc:299] Negotiation successful
0806 10:54:11.329178 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 10:54:11.329533 6813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.322450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:42745 (local address 127.30.218.1:49981)
0806 10:54:11.322691 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.322717 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:11.322906 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:11.323521 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:11.323526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:11.323578 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:11.323999 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:11.324011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.325305 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.325311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:11.327088 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:11.327105 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.328611 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.328618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:11.328823 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:11.328863 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 10:54:11.328974 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 10:54:11.329140 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":1}
I20260806 10:54:11.330360 6524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24670bda15d54efaa5bd278f2a094ab1" candidate_uuid: "3478ba34809347f3950b6b5326dd157d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef6b396ce9db43c49069415d3b6881df" is_pre_election: true
I20260806 10:54:11.330673 6807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.323295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:37991 (local address 127.30.218.3:36483)
0806 10:54:11.323517 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:11.323524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:11.323702 (+ 178us) server_negotiation.cc:407] Connection header received
0806 10:54:11.323971 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:11.323978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:11.324053 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:11.324111 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:11.324838 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.325763 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.327816 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.329072 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.329695 (+ 623us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:11.329804 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:11.330297 (+ 493us) server_negotiation.cc:299] Negotiation successful
0806 10:54:11.330433 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 10:54:11.330713 6524 raft_consensus.cc:2454] T 24670bda15d54efaa5bd278f2a094ab1 P ef6b396ce9db43c49069415d3b6881df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3478ba34809347f3950b6b5326dd157d in term 0.
I20260806 10:54:11.330823 6820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.322923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:36483 (local address 127.30.218.1:37991)
0806 10:54:11.323608 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.323623 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:11.323762 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:11.324307 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:11.324311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:11.324329 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:11.324679 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:11.324688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.325895 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.325900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:11.327630 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:11.327644 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.329964 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.329969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:11.330126 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:11.330158 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 10:54:11.330256 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 10:54:11.330412 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":21,"spinlock_wait_cycles":2304,"thread_start_us":364,"threads_started":1}
I20260806 10:54:11.331422 6304 leader_election.cc:302] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3478ba34809347f3950b6b5326dd157d, ef6b396ce9db43c49069415d3b6881df; no voters:
I20260806 10:54:11.331944 6811 raft_consensus.cc:2781] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:54:11.332113 6811 raft_consensus.cc:482] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:54:11.332229 6811 raft_consensus.cc:3037] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:11.331961 6668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24670bda15d54efaa5bd278f2a094ab1" candidate_uuid: "3478ba34809347f3950b6b5326dd157d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" is_pre_election: true
I20260806 10:54:11.332582 6668 raft_consensus.cc:2454] T 24670bda15d54efaa5bd278f2a094ab1 P b9d816dd0f9147479f65e57e4a4f33ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3478ba34809347f3950b6b5326dd157d in term 0.
I20260806 10:54:11.335006 6811 raft_consensus.cc:504] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6b396ce9db43c49069415d3b6881df" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42745 } } peers { permanent_uuid: "3478ba34809347f3950b6b5326dd157d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34939 } } peers { permanent_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 36483 } }
I20260806 10:54:11.335623 6811 leader_election.cc:288] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d [CANDIDATE]: Term 1 election: Requested vote from peers ef6b396ce9db43c49069415d3b6881df (127.30.218.2:42745), b9d816dd0f9147479f65e57e4a4f33ba (127.30.218.3:36483)
I20260806 10:54:11.336206 6524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24670bda15d54efaa5bd278f2a094ab1" candidate_uuid: "3478ba34809347f3950b6b5326dd157d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef6b396ce9db43c49069415d3b6881df"
I20260806 10:54:11.336230 6668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24670bda15d54efaa5bd278f2a094ab1" candidate_uuid: "3478ba34809347f3950b6b5326dd157d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d816dd0f9147479f65e57e4a4f33ba"
I20260806 10:54:11.336443 6524 raft_consensus.cc:3037] T 24670bda15d54efaa5bd278f2a094ab1 P ef6b396ce9db43c49069415d3b6881df [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:11.336428 6668 raft_consensus.cc:3037] T 24670bda15d54efaa5bd278f2a094ab1 P b9d816dd0f9147479f65e57e4a4f33ba [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:11.338951 6524 raft_consensus.cc:2454] T 24670bda15d54efaa5bd278f2a094ab1 P ef6b396ce9db43c49069415d3b6881df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3478ba34809347f3950b6b5326dd157d in term 1.
I20260806 10:54:11.339040 6668 raft_consensus.cc:2454] T 24670bda15d54efaa5bd278f2a094ab1 P b9d816dd0f9147479f65e57e4a4f33ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3478ba34809347f3950b6b5326dd157d in term 1.
I20260806 10:54:11.339389 6304 leader_election.cc:302] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3478ba34809347f3950b6b5326dd157d, ef6b396ce9db43c49069415d3b6881df; no voters:
I20260806 10:54:11.339762 6811 raft_consensus.cc:2781] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:54:11.340301 6811 raft_consensus.cc:686] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d [term 1 LEADER]: Becoming Leader. State: Replica: 3478ba34809347f3950b6b5326dd157d, State: Running, Role: LEADER
I20260806 10:54:11.341079 6811 consensus_queue.cc:237] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6b396ce9db43c49069415d3b6881df" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42745 } } peers { permanent_uuid: "3478ba34809347f3950b6b5326dd157d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34939 } } peers { permanent_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 36483 } }
I20260806 10:54:11.341653 6820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.341053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.30.218.1:34311)
0806 10:54:11.341248 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.341403 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260806 10:54:11.345870 6230 catalog_manager.cc:5495] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3478ba34809347f3950b6b5326dd157d (127.30.218.1). New cstate: current_term: 1 leader_uuid: "3478ba34809347f3950b6b5326dd157d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6b396ce9db43c49069415d3b6881df" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3478ba34809347f3950b6b5326dd157d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 34939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 36483 } health_report { overall_health: UNKNOWN } } }
I20260806 10:54:11.371634 6805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.365638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58738 (local address 127.30.218.1:34939)
0806 10:54:11.365896 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:11.365903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:11.365926 (+ 23us) server_negotiation.cc:407] Connection header received
0806 10:54:11.366098 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:11.366103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:11.366217 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:11.366370 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:54:11.366803 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.367609 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.369021 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.369538 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.369663 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:11.370145 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:54:11.370914 (+ 769us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:54:11.371079 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:11.371180 (+ 101us) server_negotiation.cc:299] Negotiation successful
0806 10:54:11.371291 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 10:54:11.376976 6524 raft_consensus.cc:1261] T 24670bda15d54efaa5bd278f2a094ab1 P ef6b396ce9db43c49069415d3b6881df [term 1 FOLLOWER]: Refusing update from remote peer 3478ba34809347f3950b6b5326dd157d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:54:11.377338 6668 raft_consensus.cc:1261] T 24670bda15d54efaa5bd278f2a094ab1 P b9d816dd0f9147479f65e57e4a4f33ba [term 1 FOLLOWER]: Refusing update from remote peer 3478ba34809347f3950b6b5326dd157d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:54:11.377895 6811 consensus_queue.cc:1035] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6b396ce9db43c49069415d3b6881df" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:54:11.378213 6821 consensus_queue.cc:1035] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d816dd0f9147479f65e57e4a4f33ba" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 36483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:54:11.381958 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.381491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.30.218.2:42891)
0806 10:54:11.381665 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.381775 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 10:54:11.383034 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.382397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.30.218.3:50923)
0806 10:54:11.382591 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.382731 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 10:54:11.388638 6820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.387976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.30.218.1:41149)
0806 10:54:11.388289 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.388402 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
I20260806 10:54:11.389865 6820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.389349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.30.218.1:57979)
0806 10:54:11.389579 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.389688 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260806 10:54:11.392634 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.391259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.30.218.3:46545)
0806 10:54:11.392338 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.392444 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":959}
I20260806 10:54:11.394843 6826 mvcc.cc:205] Tried to move back new op lower bound from 7315511916030828544 to 7315511915898843136. Current Snapshot: MvccSnapshot[applied={T|T < 7315511916030828544}]
I20260806 10:54:11.396003 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.395398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.30.218.2:36531)
0806 10:54:11.395681 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.395790 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260806 10:54:11.407682 6231 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:54:11.408082 6231 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:54:11.415478 6231 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b65bc5a92ba4967bf17cef40391bb14: Sending DeleteTablet for 3 replicas of tablet 24670bda15d54efaa5bd278f2a094ab1
I20260806 10:54:11.417057 6349 tablet_service.cc:1514] Processing DeleteTablet for tablet 24670bda15d54efaa5bd278f2a094ab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:54:11 UTC) from {username='oryx', principal='oryx/127.30.218.60@KRBTEST.COM'} at 127.0.0.1:58730
I20260806 10:54:11.417124 6503 tablet_service.cc:1514] Processing DeleteTablet for tablet 24670bda15d54efaa5bd278f2a094ab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:54:11 UTC) from {username='oryx', principal='oryx/127.30.218.60@KRBTEST.COM'} at 127.0.0.1:35812
I20260806 10:54:11.418063 6834 tablet_replica.cc:330] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d: stopping tablet replica
I20260806 10:54:11.418195 6647 tablet_service.cc:1514] Processing DeleteTablet for tablet 24670bda15d54efaa5bd278f2a094ab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:54:11 UTC) from {username='oryx', principal='oryx/127.30.218.60@KRBTEST.COM'} at 127.0.0.1:53690
I20260806 10:54:11.418506 6834 raft_consensus.cc:2229] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:54:11.419410 6834 raft_consensus.cc:2258] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:54:11.421195 6834 ts_tablet_manager.cc:1905] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:54:11.422070 6835 tablet_replica.cc:330] T 24670bda15d54efaa5bd278f2a094ab1 P ef6b396ce9db43c49069415d3b6881df: stopping tablet replica
I20260806 10:54:11.422389 6835 raft_consensus.cc:2229] T 24670bda15d54efaa5bd278f2a094ab1 P ef6b396ce9db43c49069415d3b6881df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:54:11.422904 6835 raft_consensus.cc:2258] T 24670bda15d54efaa5bd278f2a094ab1 P ef6b396ce9db43c49069415d3b6881df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:54:11.424156 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 6269
I20260806 10:54:11.424873 6835 ts_tablet_manager.cc:1905] T 24670bda15d54efaa5bd278f2a094ab1 P ef6b396ce9db43c49069415d3b6881df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:54:11.431946 6834 ts_tablet_manager.cc:1918] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:54:11.432304 6834 log.cc:1196] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-0/wal/wals/24670bda15d54efaa5bd278f2a094ab1
I20260806 10:54:11.432940 6835 ts_tablet_manager.cc:1918] T 24670bda15d54efaa5bd278f2a094ab1 P ef6b396ce9db43c49069415d3b6881df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:54:11.433132 6835 log.cc:1196] T 24670bda15d54efaa5bd278f2a094ab1 P ef6b396ce9db43c49069415d3b6881df: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/24670bda15d54efaa5bd278f2a094ab1
I20260806 10:54:11.433023 6834 ts_tablet_manager.cc:1939] T 24670bda15d54efaa5bd278f2a094ab1 P 3478ba34809347f3950b6b5326dd157d: Deleting consensus metadata
I20260806 10:54:11.435283 6215 catalog_manager.cc:4841] TS 3478ba34809347f3950b6b5326dd157d (127.30.218.1:34939): tablet 24670bda15d54efaa5bd278f2a094ab1 (table test-table [id=b14cf414e53541bca2bb6cbe258fac96]) successfully deleted
I20260806 10:54:11.436115 6835 ts_tablet_manager.cc:1939] T 24670bda15d54efaa5bd278f2a094ab1 P ef6b396ce9db43c49069415d3b6881df: Deleting consensus metadata
I20260806 10:54:11.440318 6836 tablet_replica.cc:330] T 24670bda15d54efaa5bd278f2a094ab1 P b9d816dd0f9147479f65e57e4a4f33ba: stopping tablet replica
I20260806 10:54:11.440621 6836 raft_consensus.cc:2229] T 24670bda15d54efaa5bd278f2a094ab1 P b9d816dd0f9147479f65e57e4a4f33ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:54:11.441089 6836 raft_consensus.cc:2258] T 24670bda15d54efaa5bd278f2a094ab1 P b9d816dd0f9147479f65e57e4a4f33ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:54:11.446914 6836 ts_tablet_manager.cc:1905] T 24670bda15d54efaa5bd278f2a094ab1 P b9d816dd0f9147479f65e57e4a4f33ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:54:11.450938 6218 catalog_manager.cc:4841] TS ef6b396ce9db43c49069415d3b6881df (127.30.218.2:42745): tablet 24670bda15d54efaa5bd278f2a094ab1 (table test-table [id=b14cf414e53541bca2bb6cbe258fac96]) successfully deleted
I20260806 10:54:11.455251 6836 ts_tablet_manager.cc:1918] T 24670bda15d54efaa5bd278f2a094ab1 P b9d816dd0f9147479f65e57e4a4f33ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:54:11.455461 6836 log.cc:1196] T 24670bda15d54efaa5bd278f2a094ab1 P b9d816dd0f9147479f65e57e4a4f33ba: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/24670bda15d54efaa5bd278f2a094ab1
I20260806 10:54:11.456032 6836 ts_tablet_manager.cc:1939] T 24670bda15d54efaa5bd278f2a094ab1 P b9d816dd0f9147479f65e57e4a4f33ba: Deleting consensus metadata
I20260806 10:54:11.457561 6218 catalog_manager.cc:4841] TS b9d816dd0f9147479f65e57e4a4f33ba (127.30.218.3:36483): tablet 24670bda15d54efaa5bd278f2a094ab1 (table test-table [id=b14cf414e53541bca2bb6cbe258fac96]) successfully deleted
I20260806 10:54:11.458467 6805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.425603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58750 (local address 127.30.218.1:34939)
0806 10:54:11.425829 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:11.425836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:11.439137 (+ 13301us) server_negotiation.cc:407] Connection header received
0806 10:54:11.439774 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:11.439781 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:11.439970 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:11.440095 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:11.442187 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.443087 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.444447 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.444937 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.445058 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:11.448711 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:11.448732 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:11.448760 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:11.448809 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:11.453603 (+ 4794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:11.454767 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:11.454782 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:11.454786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:11.455038 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:11.455691 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:11.455698 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:11.455702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:11.456364 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:11.456762 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:11.457617 (+ 855us) server_negotiation.cc:299] Negotiation successful
0806 10:54:11.457989 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 10:54:11.459308 6410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:11.578856 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 6269
I20260806 10:54:11.605916 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 6429
I20260806 10:54:11.620793 6804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.606921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35816 (local address 127.30.218.2:42745)
0806 10:54:11.607288 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:11.607295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:11.607320 (+ 25us) server_negotiation.cc:407] Connection header received
0806 10:54:11.607456 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:11.607461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:11.607554 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:11.607646 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:11.609091 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.610159 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.611480 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.611895 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.611986 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:11.614705 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:11.614724 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:11.614746 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:11.614787 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:11.618177 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:11.618816 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:11.618825 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:11.618828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:11.618985 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:11.619288 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:11.619291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:11.619293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:11.619748 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:11.620042 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:11.620259 (+ 217us) server_negotiation.cc:299] Negotiation successful
0806 10:54:11.620481 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
I20260806 10:54:11.621752 6564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:11.726212 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 6429
I20260806 10:54:11.749102 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 6582
I20260806 10:54:11.764663 6807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.750070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53694 (local address 127.30.218.3:36483)
0806 10:54:11.750288 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:11.750295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:11.750383 (+ 88us) server_negotiation.cc:407] Connection header received
0806 10:54:11.750619 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:11.750626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:11.750710 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:11.750806 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:11.752457 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.753271 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.754830 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:11.755289 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:11.755415 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:11.758630 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:11.758650 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:11.758674 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:11.758721 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:11.761885 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:11.762545 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:11.762554 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:11.762556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:11.762704 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:11.763046 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:11.763049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:11.763051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:11.763501 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:11.763834 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:11.764112 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 10:54:11.764337 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 10:54:11.765602 6708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:11.768985 6842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.768119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.0.0.1:36124)
0806 10:54:11.768625 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.768745 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":254,"threads_started":1}
I20260806 10:54:11.770666 6842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:11.770233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34161 (local address 127.0.0.1:36140)
0806 10:54:11.770391 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:11.770500 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 10:54:11.907723 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 6582
I20260806 10:54:11.935436 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 5897
I20260806 10:54:11.937101 6068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:12.081578 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 5897
I20260806 10:54:12.107223 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 5999
I20260806 10:54:12.108749 6262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:12.253211 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 5999
2026-08-06T10:54:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10131 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:54:12 dist-test-slave-49h3 krb5kdc[6848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:54:12 dist-test-slave-49h3 krb5kdc[6848](info): set up 2 sockets
Aug 06 10:54:12 dist-test-slave-49h3 krb5kdc[6848](info): commencing operation
krb5kdc: starting...
W20260806 10:54:13.347569 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:54:13 dist-test-slave-49h3 krb5kdc[6848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:54:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:54:13Z Disabled control of system clock
WARNING: no policy specified for oryx/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.62.keytab.
Entry for principal oryx/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.62.keytab.
I20260806 10:54:13.646499 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:42985
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.62.keytab
--principal=oryx/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:42985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:54:14.006551 6861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:14.006896 6861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:14.007021 6861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:14.014775 6861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:54:14.014940 6861 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 10:54:14.015010 6861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:14.015101 6861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:14.015166 6861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:54:14.015247 6861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:54:14.026103 6861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:42985
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.62.keytab
--principal=oryx/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:42985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:14.027554 6861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:14.028980 6861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:14.041779 6866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:14.041816 6867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:14.041839 6869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:14.042657 6861 server_base.cc:1034] running on GCE node
I20260806 10:54:14.043538 6861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:14.045133 6861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:14.046361 6861 hybrid_clock.cc:648] HybridClock initialized: now 1786013654046254 us; error 86 us; skew 500 ppm
I20260806 10:54:14.046727 6861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:14 dist-test-slave-49h3 krb5kdc[6848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013654, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:14.050297 6861 init.cc:377] Logged in from keytab as oryx/127.30.218.62@KRBTEST.COM (short username oryx)
I20260806 10:54:14.052016 6861 webserver.cc:460] Webserver started at http://127.30.218.62:33735/ using document root <none> and password file <none>
I20260806 10:54:14.052796 6861 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:14.052896 6861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:14.053220 6861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:14.056707 6861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "942053e138764c018d351c6936d86dd9"
format_stamp: "Formatted at 2026-08-06 10:54:14 on dist-test-slave-49h3"
server_key: "0772a055534e168734a701eefb0b22ce"
server_key_iv: "9d29f755afc183517a54cad0fde763d6"
server_key_version: "encryptionkey@0"
I20260806 10:54:14.057722 6861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "942053e138764c018d351c6936d86dd9"
format_stamp: "Formatted at 2026-08-06 10:54:14 on dist-test-slave-49h3"
server_key: "0772a055534e168734a701eefb0b22ce"
server_key_iv: "9d29f755afc183517a54cad0fde763d6"
server_key_version: "encryptionkey@0"
I20260806 10:54:14.063694 6861 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260806 10:54:14.067746 6877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:14.069308 6861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:54:14.069546 6861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "942053e138764c018d351c6936d86dd9"
format_stamp: "Formatted at 2026-08-06 10:54:14 on dist-test-slave-49h3"
server_key: "0772a055534e168734a701eefb0b22ce"
server_key_iv: "9d29f755afc183517a54cad0fde763d6"
server_key_version: "encryptionkey@0"
I20260806 10:54:14.069800 6861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:14.086817 6861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:14.091933 6861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:14.092247 6861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:14.114392 6861 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:42985
I20260806 10:54:14.114413 6938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:42985 every 8 connection(s)
I20260806 10:54:14.115938 6861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:54:14.117513 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 6861
I20260806 10:54:14.117779 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:54:14.118608 31592 external_mini_cluster.cc:1428] Setting key 2d588a7f79643cad1e8d2bc4d12108e4
I20260806 10:54:14.121399 6939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 10:54:14 dist-test-slave-49h3 krb5kdc[6848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM
I20260806 10:54:14.137730 6939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9: Bootstrap starting.
I20260806 10:54:14.141633 6939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:14.143131 6939 log.cc:824] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9: Log is configured to *not* fsync() on all Append() calls
I20260806 10:54:14.146401 6942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:14.120977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43634 (local address 127.30.218.62:42985)
0806 10:54:14.121880 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:14.121894 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:14.121939 (+ 45us) server_negotiation.cc:407] Connection header received
0806 10:54:14.123663 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:14.123695 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:14.124120 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:14.124496 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:14.126634 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:14.128109 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:14.130107 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:14.130960 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:14.131253 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:14.137715 (+ 6462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:14.137726 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:14.137744 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:14.137794 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:14.140997 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:14.141966 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:14.141978 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:14.141993 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:14.142270 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:14.142759 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:14.142764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:14.142767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:14.143580 (+ 813us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:14.143958 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:14.144437 (+ 479us) server_negotiation.cc:299] Negotiation successful
0806 10:54:14.144902 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":440,"threads_started":1}
I20260806 10:54:14.147125 6939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9: No bootstrap required, opened a new log
I20260806 10:54:14.155572 6939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942053e138764c018d351c6936d86dd9" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42985 } }
I20260806 10:54:14.156014 6939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:14.156133 6939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 942053e138764c018d351c6936d86dd9, State: Initialized, Role: FOLLOWER
I20260806 10:54:14.156757 6939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942053e138764c018d351c6936d86dd9" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42985 } }
I20260806 10:54:14.156992 6939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:54:14.157138 6939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:54:14.157307 6939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:14.159828 6939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942053e138764c018d351c6936d86dd9" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42985 } }
I20260806 10:54:14.160403 6939 leader_election.cc:302] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 942053e138764c018d351c6936d86dd9; no voters:
I20260806 10:54:14.161018 6939 leader_election.cc:288] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:54:14.161181 6944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:54:14.161999 6944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9 [term 1 LEADER]: Becoming Leader. State: Replica: 942053e138764c018d351c6936d86dd9, State: Running, Role: LEADER
I20260806 10:54:14.162575 6944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942053e138764c018d351c6936d86dd9" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42985 } }
I20260806 10:54:14.163259 6939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:54:14.166040 6946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 942053e138764c018d351c6936d86dd9. Latest consensus state: current_term: 1 leader_uuid: "942053e138764c018d351c6936d86dd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942053e138764c018d351c6936d86dd9" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42985 } } }
I20260806 10:54:14.166078 6945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "942053e138764c018d351c6936d86dd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942053e138764c018d351c6936d86dd9" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42985 } } }
I20260806 10:54:14.166649 6945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9 [sys.catalog]: This master's current role is: LEADER
I20260806 10:54:14.166656 6946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9 [sys.catalog]: This master's current role is: LEADER
W20260806 10:54:14.175398 6961 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 10:54:14.175547 6961 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 10:54:14.175697 6952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:54:14.178114 6952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:54:14.188488 6952 catalog_manager.cc:1347] Generated new cluster ID: 8ec62562038948ca8d9b8714ea58f20d
I20260806 10:54:14.188652 6952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:54:14.208313 6952 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:54:14.209785 6952 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:54:14.217836 6952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9: Generated new TSK 0
I20260806 10:54:14.218768 6952 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.30.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.1.keytab.
Entry for principal oryx/127.30.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.1.keytab.
I20260806 10:54:14.293067 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:42985
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.1.keytab
--principal=oryx/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:54:14.648712 6966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:14.649034 6966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:14.649154 6966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:14.657011 6966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:14.657161 6966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:14.657251 6966 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 10:54:14.657336 6966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:54:14.667956 6966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.1.keytab
--principal=oryx/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.218.62:42985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:14.669647 6966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:14.671159 6966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:14.685352 6971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:14.685279 6972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:14.685241 6974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:14.687286 6966 server_base.cc:1034] running on GCE node
I20260806 10:54:14.688005 6966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:14.689059 6966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:14.690315 6966 hybrid_clock.cc:648] HybridClock initialized: now 1786013654690232 us; error 51 us; skew 500 ppm
I20260806 10:54:14.690672 6966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:14 dist-test-slave-49h3 krb5kdc[6848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013654, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:14.694211 6966 init.cc:377] Logged in from keytab as oryx/127.30.218.1@KRBTEST.COM (short username oryx)
I20260806 10:54:14.695935 6966 webserver.cc:460] Webserver started at http://127.30.218.1:37143/ using document root <none> and password file <none>
I20260806 10:54:14.696729 6966 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:14.696831 6966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:14.697168 6966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:14.700768 6966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "ee5505bb889f40dd989938bda8954e15"
format_stamp: "Formatted at 2026-08-06 10:54:14 on dist-test-slave-49h3"
server_key: "2dad057ee705aedda01d3d6191774997"
server_key_iv: "d91a6f4012c79eabdf6234af0f76be35"
server_key_version: "encryptionkey@0"
I20260806 10:54:14.701926 6966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "ee5505bb889f40dd989938bda8954e15"
format_stamp: "Formatted at 2026-08-06 10:54:14 on dist-test-slave-49h3"
server_key: "2dad057ee705aedda01d3d6191774997"
server_key_iv: "d91a6f4012c79eabdf6234af0f76be35"
server_key_version: "encryptionkey@0"
I20260806 10:54:14.708779 6966 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 10:54:14.713048 6982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:14.714594 6966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:54:14.714807 6966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "ee5505bb889f40dd989938bda8954e15"
format_stamp: "Formatted at 2026-08-06 10:54:14 on dist-test-slave-49h3"
server_key: "2dad057ee705aedda01d3d6191774997"
server_key_iv: "d91a6f4012c79eabdf6234af0f76be35"
server_key_version: "encryptionkey@0"
I20260806 10:54:14.715075 6966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:14.735294 6966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:14.739674 6966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:14.739985 6966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:14.743044 6966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:14.743175 6966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:14.743325 6966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:14.743393 6966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 10:54:14 dist-test-slave-49h3 krb5kdc[6848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013654, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM
I20260806 10:54:14.770047 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:14.743882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:47055 (local address 127.30.218.62:42985)
0806 10:54:14.744382 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:14.744387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:14.745712 (+ 1325us) server_negotiation.cc:407] Connection header received
0806 10:54:14.748022 (+ 2310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:14.748028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:14.748138 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:14.748258 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:14.751437 (+ 3179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:14.752366 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:14.755603 (+ 3237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:14.756152 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:14.756301 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:14.762461 (+ 6160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:14.762471 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:14.762476 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:14.762529 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:14.765541 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:14.766718 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:14.766725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:14.766729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:14.766919 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:14.767609 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:14.767613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:14.767615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:14.767947 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:14.768114 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:14.769370 (+ 1256us) server_negotiation.cc:299] Negotiation successful
0806 10:54:14.769694 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":276,"threads_started":1}
I20260806 10:54:14.771711 6992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:14.744618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42985 (local address 127.30.218.1:47055)
0806 10:54:14.745544 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:14.745591 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:14.747708 (+ 2117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:14.748723 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:14.748743 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:14.749867 (+ 1124us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:14.751140 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:14.751192 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:14.752600 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:14.752606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:14.755339 (+ 2733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:14.755370 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:14.756650 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:14.756662 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:14.756997 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:14.758124 (+ 1127us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:14.758165 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:14.762017 (+ 3852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:14.765763 (+ 3746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:14.765773 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:14.765795 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:14.766546 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:14.767079 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:14.767086 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:14.767090 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:14.767465 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:14.768096 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:14.768109 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:14.768803 (+ 694us) client_negotiation.cc:769] Sending connection context
0806 10:54:14.769225 (+ 422us) client_negotiation.cc:240] Negotiation successful
0806 10:54:14.770000 (+ 775us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":386,"threads_started":1}
I20260806 10:54:14.793275 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:14.778174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42985 (local address 127.30.218.1:41773)
0806 10:54:14.778824 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:14.778845 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:14.779038 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:14.779412 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:14.779417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:14.780087 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:14.780553 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:14.780571 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:14.781686 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:14.781696 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:14.783294 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:14.783317 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:14.784411 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:14.784416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:14.784549 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:14.785180 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:14.785199 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:14.786322 (+ 1123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:14.790597 (+ 4275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:14.790601 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:14.790604 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:14.791075 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:14.791531 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:14.791534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:14.791537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:14.791683 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:14.792288 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:14.792291 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:14.792479 (+ 188us) client_negotiation.cc:769] Sending connection context
0806 10:54:14.792600 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 10:54:14.792906 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":350,"threads_started":1}
I20260806 10:54:14.793422 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:14.778129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:41773 (local address 127.30.218.62:42985)
0806 10:54:14.778357 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:14.778364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:14.778942 (+ 578us) server_negotiation.cc:407] Connection header received
0806 10:54:14.779187 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:14.779193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:14.779269 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:14.779355 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:14.780752 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:14.781541 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:14.783493 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:14.784212 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:14.784361 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:14.786583 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:14.786592 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:14.786596 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:14.786645 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:14.790410 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:14.791206 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:14.791212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:14.791216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:14.791417 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:14.791775 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:14.791780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:14.791784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:14.792185 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:14.792367 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:14.792738 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 10:54:14.793029 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 10:54:14.798390 7104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:38295 every 8 connection(s)
I20260806 10:54:14.798390 6966 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:38295
I20260806 10:54:14.799787 6966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:54:14.802237 7105 heartbeater.cc:347] Connected to a master server at 127.30.218.62:42985
I20260806 10:54:14.802507 7105 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:14.803262 7105 heartbeater.cc:511] Master 127.30.218.62:42985 requested a full tablet report, sending...
I20260806 10:54:14.804484 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 6966
I20260806 10:54:14.804697 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:54:14.805328 31592 external_mini_cluster.cc:1428] Setting key 07872f54cd2f84f78a37174bbb5d63bd
I20260806 10:54:14.805747 6894 ts_manager.cc:194] Registered new tserver with Master: ee5505bb889f40dd989938bda8954e15 (127.30.218.1:38295)
I20260806 10:54:14.807560 6894 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.218.1@KRBTEST.COM'} at 127.30.218.1:47055
WARNING: no policy specified for oryx/127.30.218.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.2.keytab.
Entry for principal oryx/127.30.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.2.keytab.
I20260806 10:54:14.866799 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:42985
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.2.keytab
--principal=oryx/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:54:15.219503 7109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:15.219836 7109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:15.219990 7109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:15.227571 7109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:15.227725 7109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:15.227856 7109 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 10:54:15.227977 7109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:54:15.238492 7109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.2.keytab
--principal=oryx/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.218.62:42985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:15.239974 7109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:15.241725 7109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:15.255889 7115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:15.255846 7114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:15.255928 7117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:15.256774 7109 server_base.cc:1034] running on GCE node
I20260806 10:54:15.257498 7109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:15.258527 7109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:15.259768 7109 hybrid_clock.cc:648] HybridClock initialized: now 1786013655259684 us; error 54 us; skew 500 ppm
I20260806 10:54:15.260118 7109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:15 dist-test-slave-49h3 krb5kdc[6848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013655, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:15.263598 7109 init.cc:377] Logged in from keytab as oryx/127.30.218.2@KRBTEST.COM (short username oryx)
I20260806 10:54:15.265199 7109 webserver.cc:460] Webserver started at http://127.30.218.2:34703/ using document root <none> and password file <none>
I20260806 10:54:15.265993 7109 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:15.266121 7109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:15.266422 7109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:15.269794 7109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "ce7b4388b4584f4a912c49d57cdafc40"
format_stamp: "Formatted at 2026-08-06 10:54:15 on dist-test-slave-49h3"
server_key: "21e62532dbc100be55f50e07ac96f712"
server_key_iv: "0fe2ccb41f4a350923075b9037e19bba"
server_key_version: "encryptionkey@0"
I20260806 10:54:15.270653 7109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "ce7b4388b4584f4a912c49d57cdafc40"
format_stamp: "Formatted at 2026-08-06 10:54:15 on dist-test-slave-49h3"
server_key: "21e62532dbc100be55f50e07ac96f712"
server_key_iv: "0fe2ccb41f4a350923075b9037e19bba"
server_key_version: "encryptionkey@0"
I20260806 10:54:15.276827 7109 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 10:54:15.281075 7124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:15.282647 7109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:54:15.282831 7109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "ce7b4388b4584f4a912c49d57cdafc40"
format_stamp: "Formatted at 2026-08-06 10:54:15 on dist-test-slave-49h3"
server_key: "21e62532dbc100be55f50e07ac96f712"
server_key_iv: "0fe2ccb41f4a350923075b9037e19bba"
server_key_version: "encryptionkey@0"
I20260806 10:54:15.283056 7109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:15.315198 7109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:15.319851 7109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:15.320109 7109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:15.323005 7109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:15.323112 7109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:15.323292 7109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:15.323369 7109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 10:54:15 dist-test-slave-49h3 krb5kdc[6848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013655, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM
I20260806 10:54:15.349738 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:15.323510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:56511 (local address 127.30.218.62:42985)
0806 10:54:15.324281 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:15.324288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:15.325421 (+ 1133us) server_negotiation.cc:407] Connection header received
0806 10:54:15.327772 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:15.327778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:15.327891 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:15.328010 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:15.331011 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:15.332157 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:15.334694 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:15.335329 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:15.335455 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:15.341750 (+ 6295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:15.341758 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:15.341762 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:15.341811 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:15.345202 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:15.346668 (+ 1466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:15.346675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:15.346679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:15.346869 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:15.347501 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:15.347507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:15.347510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:15.347948 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:15.348135 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:15.348943 (+ 808us) server_negotiation.cc:299] Negotiation successful
0806 10:54:15.349241 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":514,"threads_started":1}
I20260806 10:54:15.350508 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:15.324271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42985 (local address 127.30.218.2:56511)
0806 10:54:15.325209 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:15.325262 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:15.327421 (+ 2159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:15.328476 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:15.328493 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:15.329436 (+ 943us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:15.330697 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:15.330741 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:15.332344 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:15.332349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:15.334477 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:15.334501 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:15.335614 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:15.335623 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:15.335959 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:15.336890 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:15.336931 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:15.341178 (+ 4247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:15.345495 (+ 4317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:15.345510 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:15.345530 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:15.346426 (+ 896us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:15.347016 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:15.347022 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:15.347026 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:15.347352 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:15.348123 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:15.348130 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:15.348545 (+ 415us) client_negotiation.cc:769] Sending connection context
0806 10:54:15.348792 (+ 247us) client_negotiation.cc:240] Negotiation successful
0806 10:54:15.349247 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":397,"threads_started":1}
I20260806 10:54:15.372699 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:15.356305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:39299 (local address 127.30.218.62:42985)
0806 10:54:15.356509 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:15.356515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:15.357433 (+ 918us) server_negotiation.cc:407] Connection header received
0806 10:54:15.357685 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:15.357689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:15.357762 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:15.357862 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:15.359532 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:15.360278 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:15.362491 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:15.363017 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:15.363130 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:15.365920 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:15.365925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:15.365929 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:15.365967 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:15.369169 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:15.370269 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:15.370276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:15.370280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:15.370464 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:15.370964 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:15.370971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:15.370974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:15.371465 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:15.371615 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:15.372101 (+ 486us) server_negotiation.cc:299] Negotiation successful
0806 10:54:15.372351 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 10:54:15.372903 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:15.356356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42985 (local address 127.30.218.2:39299)
0806 10:54:15.357313 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:15.357335 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:15.357533 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:15.357933 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:15.357938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:15.358740 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:15.359337 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:15.359357 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:15.360419 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:15.360425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:15.362292 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:15.362316 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:15.363277 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:15.363285 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:15.363489 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:15.364381 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:15.364407 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:15.365681 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:15.369379 (+ 3698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:15.369389 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:15.369394 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:15.370086 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:15.370611 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:15.370616 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:15.370620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:15.370829 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:15.371599 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:15.371604 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:15.371808 (+ 204us) client_negotiation.cc:769] Sending connection context
0806 10:54:15.371934 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 10:54:15.372378 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":568,"threads_started":1}
I20260806 10:54:15.374063 7109 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:39917
I20260806 10:54:15.374083 7247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:39917 every 8 connection(s)
I20260806 10:54:15.375527 7109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:54:15.377705 7248 heartbeater.cc:347] Connected to a master server at 127.30.218.62:42985
I20260806 10:54:15.377929 7248 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:15.377952 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 7109
I20260806 10:54:15.378149 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:54:15.378582 31592 external_mini_cluster.cc:1428] Setting key 0bcc0f18f1eb2a947fdf242d86bcdd38
I20260806 10:54:15.378630 7248 heartbeater.cc:511] Master 127.30.218.62:42985 requested a full tablet report, sending...
I20260806 10:54:15.380084 6893 ts_manager.cc:194] Registered new tserver with Master: ce7b4388b4584f4a912c49d57cdafc40 (127.30.218.2:39917)
I20260806 10:54:15.381069 6893 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.218.2@KRBTEST.COM'} at 127.30.218.2:56511
WARNING: no policy specified for oryx/127.30.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.3.keytab.
Entry for principal oryx/127.30.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.3.keytab.
I20260806 10:54:15.438876 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:42985
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.3.keytab
--principal=oryx/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:54:15.795567 7252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:15.795938 7252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:15.796083 7252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:15.804247 7252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:15.804420 7252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:15.804553 7252 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 10:54:15.804679 7252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:54:15.811357 7105 heartbeater.cc:503] Master 127.30.218.62:42985 was elected leader, sending a full tablet report...
I20260806 10:54:15.815541 7252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.3.keytab
--principal=oryx/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.218.62:42985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:15.817032 7252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:15.818559 7252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:15.832648 7257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:15.832648 7258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:15.832988 7260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:15.833835 7252 server_base.cc:1034] running on GCE node
I20260806 10:54:15.834522 7252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:15.835578 7252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:15.836829 7252 hybrid_clock.cc:648] HybridClock initialized: now 1786013655836757 us; error 37 us; skew 500 ppm
I20260806 10:54:15.837239 7252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:15 dist-test-slave-49h3 krb5kdc[6848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013655, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:15.840775 7252 init.cc:377] Logged in from keytab as oryx/127.30.218.3@KRBTEST.COM (short username oryx)
I20260806 10:54:15.842440 7252 webserver.cc:460] Webserver started at http://127.30.218.3:41795/ using document root <none> and password file <none>
I20260806 10:54:15.843235 7252 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:15.843338 7252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:15.843667 7252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:15.847108 7252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "f3977d8fd8724b35b3d3cf52c96534bc"
format_stamp: "Formatted at 2026-08-06 10:54:15 on dist-test-slave-49h3"
server_key: "524e12490885bc0733222fe28b5c1e4f"
server_key_iv: "7a2ad2b3fc82dc2be8f48f5b99d831ea"
server_key_version: "encryptionkey@0"
I20260806 10:54:15.848031 7252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "f3977d8fd8724b35b3d3cf52c96534bc"
format_stamp: "Formatted at 2026-08-06 10:54:15 on dist-test-slave-49h3"
server_key: "524e12490885bc0733222fe28b5c1e4f"
server_key_iv: "7a2ad2b3fc82dc2be8f48f5b99d831ea"
server_key_version: "encryptionkey@0"
I20260806 10:54:15.854295 7252 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 10:54:15.858368 7267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:15.859958 7252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:54:15.860167 7252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "f3977d8fd8724b35b3d3cf52c96534bc"
format_stamp: "Formatted at 2026-08-06 10:54:15 on dist-test-slave-49h3"
server_key: "524e12490885bc0733222fe28b5c1e4f"
server_key_iv: "7a2ad2b3fc82dc2be8f48f5b99d831ea"
server_key_version: "encryptionkey@0"
I20260806 10:54:15.860425 7252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:15.872885 7252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:15.877599 7252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:15.877892 7252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:15.880687 7252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:15.880805 7252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:15.880955 7252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:15.881022 7252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 10:54:15 dist-test-slave-49h3 krb5kdc[6848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013655, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM
I20260806 10:54:15.907171 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:15.881606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:47867 (local address 127.30.218.62:42985)
0806 10:54:15.882075 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:15.882080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:15.883428 (+ 1348us) server_negotiation.cc:407] Connection header received
0806 10:54:15.885901 (+ 2473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:15.885909 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:15.886060 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:15.886209 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:15.888633 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:15.889641 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:15.892839 (+ 3198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:15.893418 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:15.893606 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:15.899111 (+ 5505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:15.899120 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:15.899125 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:15.899182 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:15.902719 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:15.903940 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:15.903945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:15.903948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:15.904127 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:15.904949 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:15.904955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:15.904958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:15.905357 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:15.905531 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:15.906452 (+ 921us) server_negotiation.cc:299] Negotiation successful
0806 10:54:15.906724 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":222,"threads_started":1}
I20260806 10:54:15.908561 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:15.882351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42985 (local address 127.30.218.3:47867)
0806 10:54:15.883245 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:15.883298 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:15.885425 (+ 2127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:15.886476 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:15.886485 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:15.887291 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:15.888298 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:15.888344 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:15.889853 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:15.889858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:15.892605 (+ 2747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:15.892632 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:15.893751 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:15.893760 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:15.894092 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:15.894957 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:15.894993 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:15.898710 (+ 3717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:15.902911 (+ 4201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:15.902927 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:15.902953 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:15.903739 (+ 786us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:15.904336 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:15.904346 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:15.904350 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:15.904788 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:15.905557 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:15.905567 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:15.906010 (+ 443us) client_negotiation.cc:769] Sending connection context
0806 10:54:15.906316 (+ 306us) client_negotiation.cc:240] Negotiation successful
0806 10:54:15.906947 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":637,"threads_started":1}
I20260806 10:54:15.930388 7252 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:45881
I20260806 10:54:15.930444 7388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:45881 every 8 connection(s)
I20260806 10:54:15.930881 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:15.915163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:40501 (local address 127.30.218.62:42985)
0806 10:54:15.915366 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:15.915372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:15.916070 (+ 698us) server_negotiation.cc:407] Connection header received
0806 10:54:15.916401 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:15.916407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:15.916480 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:15.916537 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:15.918213 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:15.919282 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:15.920965 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:15.921416 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:15.921586 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:15.923895 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:15.923903 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:15.923907 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:15.923952 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:15.927682 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:15.928527 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:15.928532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:15.928536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:15.928725 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:15.929147 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:15.929151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:15.929154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:15.929572 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:15.929764 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:15.930337 (+ 573us) server_negotiation.cc:299] Negotiation successful
0806 10:54:15.930608 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 10:54:15.931197 7347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:15.915142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42985 (local address 127.30.218.3:40501)
0806 10:54:15.915935 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:15.915957 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:15.916215 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:15.916793 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:15.916797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:15.917536 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:15.917966 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:15.917987 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:15.919448 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:15.919452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:15.920801 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:15.920817 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:15.921656 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:15.921663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:15.921832 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:15.922489 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:15.922508 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:15.923615 (+ 1107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:15.927874 (+ 4259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:15.927881 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:15.927884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:15.928394 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:15.928859 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:15.928864 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:15.928867 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:15.929011 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:15.929727 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:15.929733 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:15.930009 (+ 276us) client_negotiation.cc:769] Sending connection context
0806 10:54:15.930217 (+ 208us) client_negotiation.cc:240] Negotiation successful
0806 10:54:15.930664 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":450,"threads_started":1}
I20260806 10:54:15.932075 7252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:54:15.934546 7389 heartbeater.cc:347] Connected to a master server at 127.30.218.62:42985
I20260806 10:54:15.934851 7389 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:15.935716 7389 heartbeater.cc:511] Master 127.30.218.62:42985 requested a full tablet report, sending...
I20260806 10:54:15.937274 6893 ts_manager.cc:194] Registered new tserver with Master: f3977d8fd8724b35b3d3cf52c96534bc (127.30.218.3:45881)
I20260806 10:54:15.938553 6893 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.218.3@KRBTEST.COM'} at 127.30.218.3:47867
I20260806 10:54:15.939311 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 7252
I20260806 10:54:15.939532 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:54:15.940117 31592 external_mini_cluster.cc:1428] Setting key 7864386322af962d190805c8a1763465
I20260806 10:54:15.944175 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 10:54:15 dist-test-slave-49h3 krb5kdc[6848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013655, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 10:54:16 dist-test-slave-49h3 krb5kdc[6848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:16 dist-test-slave-49h3 krb5kdc[6848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:16.383667 7248 heartbeater.cc:503] Master 127.30.218.62:42985 was elected leader, sending a full tablet report...
I20260806 10:54:16.942715 7389 heartbeater.cc:503] Master 127.30.218.62:42985 was elected leader, sending a full tablet report...
W20260806 10:54:17.011333 7398 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.004s user 0.000s sys 0.019s
W20260806 10:54:17.013041 7276 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:16.000967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58796 (local address 127.30.218.62:42985)
0806 10:54:16.001168 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:16.001173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:16.001514 (+ 341us) server_negotiation.cc:407] Connection header received
0806 10:54:16.001676 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:16.001680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:16.001755 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:16.001853 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:16.003284 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:16.004031 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:16.005404 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:16.005904 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:16.006010 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:17.012631 (+1006621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58796: BlockingRecv error: recv got EOF from 127.0.0.1:58796 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 06 10:54:17 dist-test-slave-49h3 krb5kdc[6848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013655, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM
I20260806 10:54:17.036396 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:17.017931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58808 (local address 127.30.218.62:42985)
0806 10:54:17.018169 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:17.018176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:17.018508 (+ 332us) server_negotiation.cc:407] Connection header received
0806 10:54:17.018727 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:17.018732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:17.018827 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:17.018930 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:17.020422 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.021269 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.022660 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.023152 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.023273 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:17.030940 (+ 7667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:17.030946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:17.030950 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:17.030994 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:17.033660 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:17.034360 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:17.034366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:17.034369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:17.034508 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:17.034869 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:17.034873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:17.034875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:17.035183 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:17.035450 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:17.035804 (+ 354us) server_negotiation.cc:299] Negotiation successful
0806 10:54:17.036067 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 10:54:17.041411 6893 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:54:17.049605 6893 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:54:17.092047 7411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:17.065989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48114 (local address 127.30.218.1:38295)
0806 10:54:17.075096 (+ 9107us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:17.075111 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:17.081825 (+ 6714us) server_negotiation.cc:407] Connection header received
0806 10:54:17.082294 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:17.082304 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:17.082681 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:17.082910 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:17.084455 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.086127 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.089202 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.090657 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.090817 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:17.091128 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:17.091452 (+ 324us) server_negotiation.cc:299] Negotiation successful
0806 10:54:17.091675 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8873,"thread_start_us":8657,"threads_started":1}
I20260806 10:54:17.092139 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:17.065607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:38295 (local address 127.0.0.1:48114)
0806 10:54:17.081572 (+ 15965us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:17.081647 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:17.081943 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:17.083292 (+ 1349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:17.083302 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:17.083348 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:17.084158 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:17.084174 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.086252 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.086257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:17.089026 (+ 2769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:17.089040 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.090896 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.090904 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:17.091123 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:17.091196 (+ 73us) client_negotiation.cc:769] Sending connection context
0806 10:54:17.091377 (+ 181us) client_negotiation.cc:240] Negotiation successful
0806 10:54:17.091629 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15497,"mutex_wait_us":2,"thread_start_us":17513,"threads_started":1}
I20260806 10:54:17.093053 7412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:17.066284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46906 (local address 127.30.218.3:45881)
0806 10:54:17.081193 (+ 14909us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:17.081205 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:17.083619 (+ 2414us) server_negotiation.cc:407] Connection header received
0806 10:54:17.084038 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:17.084047 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:17.084339 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:17.084546 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:17.085161 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.086635 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.090066 (+ 3431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.091508 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.091658 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:17.091860 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:17.092498 (+ 638us) server_negotiation.cc:299] Negotiation successful
0806 10:54:17.092699 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14648,"thread_start_us":15507,"threads_started":1}
I20260806 10:54:17.093223 7414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:17.069919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43534 (local address 127.30.218.2:39917)
0806 10:54:17.082611 (+ 12692us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:17.082622 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:17.084080 (+ 1458us) server_negotiation.cc:407] Connection header received
0806 10:54:17.084398 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:17.084406 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:17.084653 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:17.084905 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:17.085794 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.087365 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.090150 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.091482 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.091617 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:17.091860 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:17.092673 (+ 813us) server_negotiation.cc:299] Negotiation successful
0806 10:54:17.092893 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12517,"thread_start_us":12364,"threads_started":1}
I20260806 10:54:17.093540 7410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:17.065699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:45881 (local address 127.0.0.1:46906)
0806 10:54:17.083401 (+ 17702us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:17.083432 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:17.083747 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:17.084553 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:17.084557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:17.084581 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:17.084997 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:17.085008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.086898 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.086904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:17.089841 (+ 2937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:17.089859 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.092094 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.092101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:17.092295 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:17.092330 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 10:54:17.092424 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 10:54:17.093115 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17574,"mutex_wait_us":32,"thread_start_us":17919,"threads_started":1}
I20260806 10:54:17.094221 7409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:17.069590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:39917 (local address 127.0.0.1:43534)
0806 10:54:17.083933 (+ 14343us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:17.083948 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:17.084191 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:17.085162 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:17.085166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:17.085189 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:17.085578 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:17.085590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.087555 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.087560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:17.089939 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:17.089951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.092055 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.092062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:17.092409 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:17.092440 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 10:54:17.092516 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 10:54:17.093842 (+ 1326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14244,"mutex_wait_us":985,"thread_start_us":12619,"threads_started":1}
I20260806 10:54:17.097805 7318 tablet_service.cc:1467] Processing CreateTablet for tablet 311a58d92051454099f5c808c268630e (DEFAULT_TABLE table=test-table [id=640ca55913574ef9a508aca9b573393d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:54:17.097819 7176 tablet_service.cc:1467] Processing CreateTablet for tablet 311a58d92051454099f5c808c268630e (DEFAULT_TABLE table=test-table [id=640ca55913574ef9a508aca9b573393d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:54:17.097826 7033 tablet_service.cc:1467] Processing CreateTablet for tablet 311a58d92051454099f5c808c268630e (DEFAULT_TABLE table=test-table [id=640ca55913574ef9a508aca9b573393d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:54:17.099503 7318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 311a58d92051454099f5c808c268630e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:17.099507 7033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 311a58d92051454099f5c808c268630e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:17.099509 7176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 311a58d92051454099f5c808c268630e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:17.113530 7415 tablet_bootstrap.cc:492] T 311a58d92051454099f5c808c268630e P f3977d8fd8724b35b3d3cf52c96534bc: Bootstrap starting.
I20260806 10:54:17.118831 7417 tablet_bootstrap.cc:492] T 311a58d92051454099f5c808c268630e P ee5505bb889f40dd989938bda8954e15: Bootstrap starting.
I20260806 10:54:17.120141 7415 tablet_bootstrap.cc:654] T 311a58d92051454099f5c808c268630e P f3977d8fd8724b35b3d3cf52c96534bc: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:17.121829 7416 tablet_bootstrap.cc:492] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40: Bootstrap starting.
I20260806 10:54:17.121886 7415 log.cc:824] T 311a58d92051454099f5c808c268630e P f3977d8fd8724b35b3d3cf52c96534bc: Log is configured to *not* fsync() on all Append() calls
I20260806 10:54:17.124387 7417 tablet_bootstrap.cc:654] T 311a58d92051454099f5c808c268630e P ee5505bb889f40dd989938bda8954e15: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:17.126121 7416 tablet_bootstrap.cc:654] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:17.127592 7417 log.cc:824] T 311a58d92051454099f5c808c268630e P ee5505bb889f40dd989938bda8954e15: Log is configured to *not* fsync() on all Append() calls
I20260806 10:54:17.127591 7416 log.cc:824] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40: Log is configured to *not* fsync() on all Append() calls
I20260806 10:54:17.133073 7415 tablet_bootstrap.cc:492] T 311a58d92051454099f5c808c268630e P f3977d8fd8724b35b3d3cf52c96534bc: No bootstrap required, opened a new log
I20260806 10:54:17.133074 7416 tablet_bootstrap.cc:492] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40: No bootstrap required, opened a new log
I20260806 10:54:17.133081 7417 tablet_bootstrap.cc:492] T 311a58d92051454099f5c808c268630e P ee5505bb889f40dd989938bda8954e15: No bootstrap required, opened a new log
I20260806 10:54:17.133422 7416 ts_tablet_manager.cc:1397] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260806 10:54:17.133423 7415 ts_tablet_manager.cc:1397] T 311a58d92051454099f5c808c268630e P f3977d8fd8724b35b3d3cf52c96534bc: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.013s
I20260806 10:54:17.133507 7417 ts_tablet_manager.cc:1397] T 311a58d92051454099f5c808c268630e P ee5505bb889f40dd989938bda8954e15: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.008s
I20260806 10:54:17.141404 7417 raft_consensus.cc:348] T 311a58d92051454099f5c808c268630e P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } }
I20260806 10:54:17.141891 7417 raft_consensus.cc:374] T 311a58d92051454099f5c808c268630e P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:17.142009 7417 raft_consensus.cc:729] T 311a58d92051454099f5c808c268630e P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee5505bb889f40dd989938bda8954e15, State: Initialized, Role: FOLLOWER
I20260806 10:54:17.142701 7417 consensus_queue.cc:260] T 311a58d92051454099f5c808c268630e P ee5505bb889f40dd989938bda8954e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } }
I20260806 10:54:17.144809 7416 raft_consensus.cc:348] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } }
I20260806 10:54:17.144811 7415 raft_consensus.cc:348] T 311a58d92051454099f5c808c268630e P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } }
I20260806 10:54:17.145264 7415 raft_consensus.cc:374] T 311a58d92051454099f5c808c268630e P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:17.145267 7416 raft_consensus.cc:374] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:17.145365 7416 raft_consensus.cc:729] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce7b4388b4584f4a912c49d57cdafc40, State: Initialized, Role: FOLLOWER
I20260806 10:54:17.145357 7415 raft_consensus.cc:729] T 311a58d92051454099f5c808c268630e P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3977d8fd8724b35b3d3cf52c96534bc, State: Initialized, Role: FOLLOWER
I20260806 10:54:17.145947 7417 ts_tablet_manager.cc:1428] T 311a58d92051454099f5c808c268630e P ee5505bb889f40dd989938bda8954e15: Time spent starting tablet: real 0.012s user 0.007s sys 0.003s
I20260806 10:54:17.146044 7415 consensus_queue.cc:260] T 311a58d92051454099f5c808c268630e P f3977d8fd8724b35b3d3cf52c96534bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } }
I20260806 10:54:17.146690 7416 consensus_queue.cc:260] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } }
I20260806 10:54:17.148496 7415 ts_tablet_manager.cc:1428] T 311a58d92051454099f5c808c268630e P f3977d8fd8724b35b3d3cf52c96534bc: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260806 10:54:17.150226 7416 ts_tablet_manager.cc:1428] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40: Time spent starting tablet: real 0.017s user 0.010s sys 0.005s
W20260806 10:54:17.185123 7390 tablet.cc:2369] T 311a58d92051454099f5c808c268630e P f3977d8fd8724b35b3d3cf52c96534bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:54:17.202729 7423 raft_consensus.cc:482] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:54:17.203150 7423 raft_consensus.cc:504] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } }
I20260806 10:54:17.205514 7423 leader_election.cc:288] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee5505bb889f40dd989938bda8954e15 (127.30.218.1:38295), f3977d8fd8724b35b3d3cf52c96534bc (127.30.218.3:45881)
I20260806 10:54:17.212774 7412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:17.206074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:51683 (local address 127.30.218.3:45881)
0806 10:54:17.206357 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:17.206365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:17.207058 (+ 693us) server_negotiation.cc:407] Connection header received
0806 10:54:17.207295 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:17.207299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:17.207370 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:17.207457 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:17.208079 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.208863 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.210903 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.211863 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.212027 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:17.212109 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:17.212356 (+ 247us) server_negotiation.cc:299] Negotiation successful
0806 10:54:17.212508 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 10:54:17.212770 7411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:17.206173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:56171 (local address 127.30.218.1:38295)
0806 10:54:17.206409 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:17.206415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:17.207204 (+ 789us) server_negotiation.cc:407] Connection header received
0806 10:54:17.207385 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:17.207389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:17.207459 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:17.207535 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:17.208181 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.208914 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.210929 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.211820 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.211987 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:17.212073 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:17.212352 (+ 279us) server_negotiation.cc:299] Negotiation successful
0806 10:54:17.212489 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 10:54:17.212944 7425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:17.206265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:38295 (local address 127.30.218.2:56171)
0806 10:54:17.207093 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:17.207110 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:17.207247 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:17.207558 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:17.207563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:17.207590 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:17.208022 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:17.208031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.209043 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.209048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:17.210756 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:17.210768 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.212041 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.212049 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:17.212193 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:17.212220 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 10:54:17.212315 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 10:54:17.212484 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":39,"thread_start_us":485,"threads_started":1}
I20260806 10:54:17.213406 7424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:17.205928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:45881 (local address 127.30.218.2:51683)
0806 10:54:17.206860 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:17.206892 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:17.207120 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:17.207474 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:17.207479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:17.207503 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:17.207936 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:17.207947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.208990 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.208993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:17.210759 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:17.210770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.212086 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.212092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:17.212219 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:17.212240 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 10:54:17.212317 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 10:54:17.212504 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":94,"spinlock_wait_cycles":1792,"thread_start_us":520,"threads_started":1}
I20260806 10:54:17.214227 7053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "311a58d92051454099f5c808c268630e" candidate_uuid: "ce7b4388b4584f4a912c49d57cdafc40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5505bb889f40dd989938bda8954e15" is_pre_election: true
I20260806 10:54:17.214499 7342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "311a58d92051454099f5c808c268630e" candidate_uuid: "ce7b4388b4584f4a912c49d57cdafc40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" is_pre_election: true
I20260806 10:54:17.214995 7053 raft_consensus.cc:2454] T 311a58d92051454099f5c808c268630e P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce7b4388b4584f4a912c49d57cdafc40 in term 0.
I20260806 10:54:17.215071 7342 raft_consensus.cc:2454] T 311a58d92051454099f5c808c268630e P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce7b4388b4584f4a912c49d57cdafc40 in term 0.
I20260806 10:54:17.215785 7127 leader_election.cc:302] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce7b4388b4584f4a912c49d57cdafc40, ee5505bb889f40dd989938bda8954e15; no voters:
I20260806 10:54:17.216205 7423 raft_consensus.cc:2781] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:54:17.216343 7423 raft_consensus.cc:482] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:54:17.216459 7423 raft_consensus.cc:3037] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:17.219267 7423 raft_consensus.cc:504] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } }
I20260806 10:54:17.220038 7423 leader_election.cc:288] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40 [CANDIDATE]: Term 1 election: Requested vote from peers ee5505bb889f40dd989938bda8954e15 (127.30.218.1:38295), f3977d8fd8724b35b3d3cf52c96534bc (127.30.218.3:45881)
I20260806 10:54:17.220517 7342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "311a58d92051454099f5c808c268630e" candidate_uuid: "ce7b4388b4584f4a912c49d57cdafc40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3977d8fd8724b35b3d3cf52c96534bc"
I20260806 10:54:17.220566 7053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "311a58d92051454099f5c808c268630e" candidate_uuid: "ce7b4388b4584f4a912c49d57cdafc40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5505bb889f40dd989938bda8954e15"
I20260806 10:54:17.220760 7342 raft_consensus.cc:3037] T 311a58d92051454099f5c808c268630e P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:17.220815 7053 raft_consensus.cc:3037] T 311a58d92051454099f5c808c268630e P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:17.223332 7053 raft_consensus.cc:2454] T 311a58d92051454099f5c808c268630e P ee5505bb889f40dd989938bda8954e15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce7b4388b4584f4a912c49d57cdafc40 in term 1.
I20260806 10:54:17.223335 7342 raft_consensus.cc:2454] T 311a58d92051454099f5c808c268630e P f3977d8fd8724b35b3d3cf52c96534bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce7b4388b4584f4a912c49d57cdafc40 in term 1.
I20260806 10:54:17.223775 7125 leader_election.cc:302] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce7b4388b4584f4a912c49d57cdafc40, f3977d8fd8724b35b3d3cf52c96534bc; no voters:
I20260806 10:54:17.224108 7423 raft_consensus.cc:2781] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:54:17.224576 7423 raft_consensus.cc:686] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40 [term 1 LEADER]: Becoming Leader. State: Replica: ce7b4388b4584f4a912c49d57cdafc40, State: Running, Role: LEADER
I20260806 10:54:17.225169 7423 consensus_queue.cc:237] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } }
I20260806 10:54:17.230432 6893 catalog_manager.cc:5495] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce7b4388b4584f4a912c49d57cdafc40 (127.30.218.2). New cstate: current_term: 1 leader_uuid: "ce7b4388b4584f4a912c49d57cdafc40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } health_report { overall_health: HEALTHY } } }
I20260806 10:54:17.272151 7414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:17.266179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43550 (local address 127.30.218.2:39917)
0806 10:54:17.266409 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:17.266415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:17.266437 (+ 22us) server_negotiation.cc:407] Connection header received
0806 10:54:17.266692 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:17.266697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:17.266812 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:17.266964 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:54:17.267323 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.268220 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.269714 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.270164 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.270266 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:17.270749 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:54:17.271509 (+ 760us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:54:17.271629 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:17.271728 (+ 99us) server_negotiation.cc:299] Negotiation successful
0806 10:54:17.271821 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 10:54:17.282177 7342 raft_consensus.cc:1261] T 311a58d92051454099f5c808c268630e P f3977d8fd8724b35b3d3cf52c96534bc [term 1 FOLLOWER]: Refusing update from remote peer ce7b4388b4584f4a912c49d57cdafc40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:54:17.282176 7053 raft_consensus.cc:1261] T 311a58d92051454099f5c808c268630e P ee5505bb889f40dd989938bda8954e15 [term 1 FOLLOWER]: Refusing update from remote peer ce7b4388b4584f4a912c49d57cdafc40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:54:17.283257 7426 consensus_queue.cc:1035] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:54:17.283655 7423 consensus_queue.cc:1035] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:54:17.301084 7431 mvcc.cc:205] Tried to move back new op lower bound from 7315511940216233984 to 7315511939998662656. Current Snapshot: MvccSnapshot[applied={T|T < 7315511940216233984}]
I20260806 10:54:17.317531 6894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:54:17.317924 6894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:54:17.322733 6894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9: Sending DeleteTablet for 3 replicas of tablet 311a58d92051454099f5c808c268630e
I20260806 10:54:17.324105 7176 tablet_service.cc:1514] Processing DeleteTablet for tablet 311a58d92051454099f5c808c268630e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:54:17 UTC) from {username='oryx', principal='oryx/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:43534
I20260806 10:54:17.324106 7033 tablet_service.cc:1514] Processing DeleteTablet for tablet 311a58d92051454099f5c808c268630e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:54:17 UTC) from {username='oryx', principal='oryx/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:48114
I20260806 10:54:17.324098 7318 tablet_service.cc:1514] Processing DeleteTablet for tablet 311a58d92051454099f5c808c268630e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:54:17 UTC) from {username='oryx', principal='oryx/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:46906
I20260806 10:54:17.327153 6893 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:54:17.328079 6893 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:54:17.337288 7441 tablet_replica.cc:330] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40: stopping tablet replica
I20260806 10:54:17.338238 7442 tablet_replica.cc:330] T 311a58d92051454099f5c808c268630e P f3977d8fd8724b35b3d3cf52c96534bc: stopping tablet replica
I20260806 10:54:17.338891 7442 raft_consensus.cc:2229] T 311a58d92051454099f5c808c268630e P f3977d8fd8724b35b3d3cf52c96534bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:54:17.339617 7442 raft_consensus.cc:2258] T 311a58d92051454099f5c808c268630e P f3977d8fd8724b35b3d3cf52c96534bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:54:17.339558 7032 tablet_service.cc:1467] Processing CreateTablet for tablet 11635649ca354ef18e9df20cca74d307 (DEFAULT_TABLE table=test-table [id=42df7ee50db345098828214153b0e182]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:54:17.339530 7318 tablet_service.cc:1467] Processing CreateTablet for tablet 11635649ca354ef18e9df20cca74d307 (DEFAULT_TABLE table=test-table [id=42df7ee50db345098828214153b0e182]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:54:17.340272 7318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11635649ca354ef18e9df20cca74d307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:17.340297 7032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11635649ca354ef18e9df20cca74d307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:17.341512 7441 raft_consensus.cc:2229] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:54:17.342550 7441 raft_consensus.cc:2258] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:54:17.342468 7175 tablet_service.cc:1467] Processing CreateTablet for tablet 11635649ca354ef18e9df20cca74d307 (DEFAULT_TABLE table=test-table [id=42df7ee50db345098828214153b0e182]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:54:17.343195 7175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11635649ca354ef18e9df20cca74d307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:17.347060 7442 ts_tablet_manager.cc:1905] T 311a58d92051454099f5c808c268630e P f3977d8fd8724b35b3d3cf52c96534bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:54:17.348090 7415 tablet_bootstrap.cc:492] T 11635649ca354ef18e9df20cca74d307 P f3977d8fd8724b35b3d3cf52c96534bc: Bootstrap starting.
I20260806 10:54:17.350123 7415 tablet_bootstrap.cc:654] T 11635649ca354ef18e9df20cca74d307 P f3977d8fd8724b35b3d3cf52c96534bc: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:17.350142 7440 tablet_replica.cc:330] T 311a58d92051454099f5c808c268630e P ee5505bb889f40dd989938bda8954e15: stopping tablet replica
I20260806 10:54:17.350860 7440 raft_consensus.cc:2229] T 311a58d92051454099f5c808c268630e P ee5505bb889f40dd989938bda8954e15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:54:17.351619 7440 raft_consensus.cc:2258] T 311a58d92051454099f5c808c268630e P ee5505bb889f40dd989938bda8954e15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:54:17.351900 7441 ts_tablet_manager.cc:1905] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:54:17.353214 7417 tablet_bootstrap.cc:492] T 11635649ca354ef18e9df20cca74d307 P ee5505bb889f40dd989938bda8954e15: Bootstrap starting.
I20260806 10:54:17.353525 7416 tablet_bootstrap.cc:492] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40: Bootstrap starting.
I20260806 10:54:17.355281 7417 tablet_bootstrap.cc:654] T 11635649ca354ef18e9df20cca74d307 P ee5505bb889f40dd989938bda8954e15: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:17.355348 7416 tablet_bootstrap.cc:654] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:17.364106 7441 ts_tablet_manager.cc:1918] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:54:17.364404 7441 log.cc:1196] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/311a58d92051454099f5c808c268630e
I20260806 10:54:17.365939 7417 tablet_bootstrap.cc:492] T 11635649ca354ef18e9df20cca74d307 P ee5505bb889f40dd989938bda8954e15: No bootstrap required, opened a new log
I20260806 10:54:17.366144 7417 ts_tablet_manager.cc:1397] T 11635649ca354ef18e9df20cca74d307 P ee5505bb889f40dd989938bda8954e15: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.001s
I20260806 10:54:17.366199 7440 ts_tablet_manager.cc:1905] T 311a58d92051454099f5c808c268630e P ee5505bb889f40dd989938bda8954e15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:54:17.366663 7415 tablet_bootstrap.cc:492] T 11635649ca354ef18e9df20cca74d307 P f3977d8fd8724b35b3d3cf52c96534bc: No bootstrap required, opened a new log
I20260806 10:54:17.366729 7442 ts_tablet_manager.cc:1918] T 311a58d92051454099f5c808c268630e P f3977d8fd8724b35b3d3cf52c96534bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:54:17.366917 7442 log.cc:1196] T 311a58d92051454099f5c808c268630e P f3977d8fd8724b35b3d3cf52c96534bc: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/311a58d92051454099f5c808c268630e
I20260806 10:54:17.366968 7415 ts_tablet_manager.cc:1397] T 11635649ca354ef18e9df20cca74d307 P f3977d8fd8724b35b3d3cf52c96534bc: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.008s
I20260806 10:54:17.367280 7417 raft_consensus.cc:348] T 11635649ca354ef18e9df20cca74d307 P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:17.367570 7442 ts_tablet_manager.cc:1939] T 311a58d92051454099f5c808c268630e P f3977d8fd8724b35b3d3cf52c96534bc: Deleting consensus metadata
I20260806 10:54:17.367586 7417 raft_consensus.cc:374] T 11635649ca354ef18e9df20cca74d307 P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:17.367694 7417 raft_consensus.cc:729] T 11635649ca354ef18e9df20cca74d307 P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee5505bb889f40dd989938bda8954e15, State: Initialized, Role: FOLLOWER
I20260806 10:54:17.368276 7415 raft_consensus.cc:348] T 11635649ca354ef18e9df20cca74d307 P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:17.368593 7415 raft_consensus.cc:374] T 11635649ca354ef18e9df20cca74d307 P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:17.368742 7415 raft_consensus.cc:729] T 11635649ca354ef18e9df20cca74d307 P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3977d8fd8724b35b3d3cf52c96534bc, State: Initialized, Role: FOLLOWER
I20260806 10:54:17.369000 7441 ts_tablet_manager.cc:1939] T 311a58d92051454099f5c808c268630e P ce7b4388b4584f4a912c49d57cdafc40: Deleting consensus metadata
I20260806 10:54:17.369179 7415 consensus_queue.cc:260] T 11635649ca354ef18e9df20cca74d307 P f3977d8fd8724b35b3d3cf52c96534bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:17.369532 7417 consensus_queue.cc:260] T 11635649ca354ef18e9df20cca74d307 P ee5505bb889f40dd989938bda8954e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:17.370280 7417 ts_tablet_manager.cc:1428] T 11635649ca354ef18e9df20cca74d307 P ee5505bb889f40dd989938bda8954e15: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260806 10:54:17.371507 6881 catalog_manager.cc:4841] TS ce7b4388b4584f4a912c49d57cdafc40 (127.30.218.2:39917): tablet 311a58d92051454099f5c808c268630e (table test-table [id=640ca55913574ef9a508aca9b573393d]) successfully deleted
I20260806 10:54:17.372069 7416 tablet_bootstrap.cc:492] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40: No bootstrap required, opened a new log
I20260806 10:54:17.372241 7416 ts_tablet_manager.cc:1397] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.000s
I20260806 10:54:17.373293 7416 raft_consensus.cc:348] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:17.373596 7416 raft_consensus.cc:374] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:17.373749 7416 raft_consensus.cc:729] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce7b4388b4584f4a912c49d57cdafc40, State: Initialized, Role: FOLLOWER
I20260806 10:54:17.374132 7416 consensus_queue.cc:260] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:17.375747 7416 ts_tablet_manager.cc:1428] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 10:54:17.377774 7440 ts_tablet_manager.cc:1918] T 311a58d92051454099f5c808c268630e P ee5505bb889f40dd989938bda8954e15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:54:17.377861 6878 catalog_manager.cc:4841] TS f3977d8fd8724b35b3d3cf52c96534bc (127.30.218.3:45881): tablet 311a58d92051454099f5c808c268630e (table test-table [id=640ca55913574ef9a508aca9b573393d]) successfully deleted
I20260806 10:54:17.377985 7440 log.cc:1196] T 311a58d92051454099f5c808c268630e P ee5505bb889f40dd989938bda8954e15: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-0/wal/wals/311a58d92051454099f5c808c268630e
I20260806 10:54:17.379024 7440 ts_tablet_manager.cc:1939] T 311a58d92051454099f5c808c268630e P ee5505bb889f40dd989938bda8954e15: Deleting consensus metadata
W20260806 10:54:17.379702 7249 tablet.cc:2369] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:54:17.381034 7415 ts_tablet_manager.cc:1428] T 11635649ca354ef18e9df20cca74d307 P f3977d8fd8724b35b3d3cf52c96534bc: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20260806 10:54:17.381371 6880 catalog_manager.cc:4841] TS ee5505bb889f40dd989938bda8954e15 (127.30.218.1:38295): tablet 311a58d92051454099f5c808c268630e (table test-table [id=640ca55913574ef9a508aca9b573393d]) successfully deleted
W20260806 10:54:17.554874 7106 tablet.cc:2369] T 11635649ca354ef18e9df20cca74d307 P ee5505bb889f40dd989938bda8954e15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:54:17.568012 7423 raft_consensus.cc:482] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:54:17.568296 7423 raft_consensus.cc:504] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:17.569263 7423 leader_election.cc:288] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee5505bb889f40dd989938bda8954e15 (127.30.218.1:38295), f3977d8fd8724b35b3d3cf52c96534bc (127.30.218.3:45881)
I20260806 10:54:17.570020 7053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11635649ca354ef18e9df20cca74d307" candidate_uuid: "ce7b4388b4584f4a912c49d57cdafc40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5505bb889f40dd989938bda8954e15" is_pre_election: true
I20260806 10:54:17.570031 7342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11635649ca354ef18e9df20cca74d307" candidate_uuid: "ce7b4388b4584f4a912c49d57cdafc40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" is_pre_election: true
I20260806 10:54:17.570330 7053 raft_consensus.cc:2454] T 11635649ca354ef18e9df20cca74d307 P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce7b4388b4584f4a912c49d57cdafc40 in term 0.
I20260806 10:54:17.570370 7342 raft_consensus.cc:2454] T 11635649ca354ef18e9df20cca74d307 P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce7b4388b4584f4a912c49d57cdafc40 in term 0.
I20260806 10:54:17.570855 7127 leader_election.cc:302] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce7b4388b4584f4a912c49d57cdafc40, ee5505bb889f40dd989938bda8954e15; no voters:
I20260806 10:54:17.571178 7423 raft_consensus.cc:2781] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:54:17.571313 7423 raft_consensus.cc:482] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:54:17.571414 7423 raft_consensus.cc:3037] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:17.574240 7423 raft_consensus.cc:504] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:17.574944 7423 leader_election.cc:288] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40 [CANDIDATE]: Term 1 election: Requested vote from peers ee5505bb889f40dd989938bda8954e15 (127.30.218.1:38295), f3977d8fd8724b35b3d3cf52c96534bc (127.30.218.3:45881)
I20260806 10:54:17.575488 7053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11635649ca354ef18e9df20cca74d307" candidate_uuid: "ce7b4388b4584f4a912c49d57cdafc40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5505bb889f40dd989938bda8954e15"
I20260806 10:54:17.575500 7342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11635649ca354ef18e9df20cca74d307" candidate_uuid: "ce7b4388b4584f4a912c49d57cdafc40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3977d8fd8724b35b3d3cf52c96534bc"
I20260806 10:54:17.575706 7053 raft_consensus.cc:3037] T 11635649ca354ef18e9df20cca74d307 P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:17.575716 7342 raft_consensus.cc:3037] T 11635649ca354ef18e9df20cca74d307 P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:17.578034 7053 raft_consensus.cc:2454] T 11635649ca354ef18e9df20cca74d307 P ee5505bb889f40dd989938bda8954e15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce7b4388b4584f4a912c49d57cdafc40 in term 1.
I20260806 10:54:17.578490 7127 leader_election.cc:302] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce7b4388b4584f4a912c49d57cdafc40, ee5505bb889f40dd989938bda8954e15; no voters:
I20260806 10:54:17.578780 7423 raft_consensus.cc:2781] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:54:17.578904 7342 raft_consensus.cc:2454] T 11635649ca354ef18e9df20cca74d307 P f3977d8fd8724b35b3d3cf52c96534bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce7b4388b4584f4a912c49d57cdafc40 in term 1.
I20260806 10:54:17.578950 7423 raft_consensus.cc:686] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40 [term 1 LEADER]: Becoming Leader. State: Replica: ce7b4388b4584f4a912c49d57cdafc40, State: Running, Role: LEADER
I20260806 10:54:17.579308 7423 consensus_queue.cc:237] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:17.582682 6894 catalog_manager.cc:5495] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce7b4388b4584f4a912c49d57cdafc40 (127.30.218.2). New cstate: current_term: 1 leader_uuid: "ce7b4388b4584f4a912c49d57cdafc40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } health_report { overall_health: UNKNOWN } } }
Aug 06 10:54:17 dist-test-slave-49h3 krb5kdc[6848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013654, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.62@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM
I20260806 10:54:17.646638 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:17.628636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58824 (local address 127.30.218.62:42985)
0806 10:54:17.629412 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:17.629421 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:17.629752 (+ 331us) server_negotiation.cc:407] Connection header received
0806 10:54:17.629956 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:17.629962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:17.630058 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:17.630209 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:17.631631 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.632923 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.634570 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.635263 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.635423 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:17.639609 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:17.639617 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:17.639622 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:17.639666 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:17.643480 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:17.644341 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:17.644345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:17.644347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:17.644481 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:17.644899 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:17.644903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:17.644905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:17.645243 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:17.645390 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:17.645930 (+ 540us) server_negotiation.cc:299] Negotiation successful
0806 10:54:17.646283 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":142,"server-negotiator.queue_time_us":550,"thread_start_us":361,"threads_started":1}
I20260806 10:54:17.646646 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:17.628564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42985 (local address 127.0.0.1:58824)
0806 10:54:17.629625 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:17.629648 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:17.629822 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:17.630221 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:17.630224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:17.631218 (+ 994us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:17.631461 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:17.631471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.633100 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.633105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:17.634404 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:17.634419 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.635451 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.635456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:17.635606 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:17.636425 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:17.636446 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:17.639365 (+ 2919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:17.643634 (+ 4269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:17.643647 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:17.643655 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:17.644184 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:17.644584 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:17.644589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:17.644591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:17.644772 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:17.645380 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:17.645389 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:17.645627 (+ 238us) client_negotiation.cc:769] Sending connection context
0806 10:54:17.645810 (+ 183us) client_negotiation.cc:240] Negotiation successful
0806 10:54:17.646146 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"thread_start_us":689,"threads_started":1}
I20260806 10:54:17.653425 6894 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:58824:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 10:54:17.654783 6894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:54:17.663651 7032 tablet_service.cc:1467] Processing CreateTablet for tablet 70d908fd179f40af99986c9315c8035e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1577959c65df4616831430f2f40b4ce9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 10:54:17.664144 7175 tablet_service.cc:1467] Processing CreateTablet for tablet 70d908fd179f40af99986c9315c8035e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1577959c65df4616831430f2f40b4ce9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 10:54:17.664427 7032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70d908fd179f40af99986c9315c8035e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:17.664496 7318 tablet_service.cc:1467] Processing CreateTablet for tablet 70d908fd179f40af99986c9315c8035e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1577959c65df4616831430f2f40b4ce9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 10:54:17.664790 7175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70d908fd179f40af99986c9315c8035e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:17.665167 7318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70d908fd179f40af99986c9315c8035e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:17.673378 7416 tablet_bootstrap.cc:492] T 70d908fd179f40af99986c9315c8035e P ce7b4388b4584f4a912c49d57cdafc40: Bootstrap starting.
I20260806 10:54:17.673485 7417 tablet_bootstrap.cc:492] T 70d908fd179f40af99986c9315c8035e P ee5505bb889f40dd989938bda8954e15: Bootstrap starting.
I20260806 10:54:17.675568 7417 tablet_bootstrap.cc:654] T 70d908fd179f40af99986c9315c8035e P ee5505bb889f40dd989938bda8954e15: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:17.675588 7416 tablet_bootstrap.cc:654] T 70d908fd179f40af99986c9315c8035e P ce7b4388b4584f4a912c49d57cdafc40: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:17.678131 7417 tablet_bootstrap.cc:492] T 70d908fd179f40af99986c9315c8035e P ee5505bb889f40dd989938bda8954e15: No bootstrap required, opened a new log
I20260806 10:54:17.678321 7417 ts_tablet_manager.cc:1397] T 70d908fd179f40af99986c9315c8035e P ee5505bb889f40dd989938bda8954e15: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260806 10:54:17.679224 7417 raft_consensus.cc:348] T 70d908fd179f40af99986c9315c8035e P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:17.679512 7417 raft_consensus.cc:374] T 70d908fd179f40af99986c9315c8035e P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:17.679598 7417 raft_consensus.cc:729] T 70d908fd179f40af99986c9315c8035e P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee5505bb889f40dd989938bda8954e15, State: Initialized, Role: FOLLOWER
I20260806 10:54:17.679958 7417 consensus_queue.cc:260] T 70d908fd179f40af99986c9315c8035e P ee5505bb889f40dd989938bda8954e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:17.680486 7421 tablet_replica.cc:439] T 70d908fd179f40af99986c9315c8035e P ee5505bb889f40dd989938bda8954e15: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } } }
I20260806 10:54:17.680932 7421 tablet_replica.cc:442] T 70d908fd179f40af99986c9315c8035e P ee5505bb889f40dd989938bda8954e15: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:54:17.680979 7415 tablet_bootstrap.cc:492] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc: Bootstrap starting.
I20260806 10:54:17.681692 7417 ts_tablet_manager.cc:1428] T 70d908fd179f40af99986c9315c8035e P ee5505bb889f40dd989938bda8954e15: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260806 10:54:17.683292 7415 tablet_bootstrap.cc:654] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:17.686096 7415 tablet_bootstrap.cc:492] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc: No bootstrap required, opened a new log
I20260806 10:54:17.686290 7415 ts_tablet_manager.cc:1397] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 10:54:17.687160 7415 raft_consensus.cc:348] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:17.687456 7415 raft_consensus.cc:374] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:17.687546 7415 raft_consensus.cc:729] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3977d8fd8724b35b3d3cf52c96534bc, State: Initialized, Role: FOLLOWER
I20260806 10:54:17.687829 7415 consensus_queue.cc:260] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:17.688494 7415 ts_tablet_manager.cc:1428] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260806 10:54:17.688362 7422 tablet_replica.cc:439] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } } }
I20260806 10:54:17.688777 7422 tablet_replica.cc:442] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:54:17.692008 7416 tablet_bootstrap.cc:492] T 70d908fd179f40af99986c9315c8035e P ce7b4388b4584f4a912c49d57cdafc40: No bootstrap required, opened a new log
I20260806 10:54:17.692246 7416 ts_tablet_manager.cc:1397] T 70d908fd179f40af99986c9315c8035e P ce7b4388b4584f4a912c49d57cdafc40: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.002s
I20260806 10:54:17.693231 7416 raft_consensus.cc:348] T 70d908fd179f40af99986c9315c8035e P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:17.693573 7416 raft_consensus.cc:374] T 70d908fd179f40af99986c9315c8035e P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:17.693675 7416 raft_consensus.cc:729] T 70d908fd179f40af99986c9315c8035e P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce7b4388b4584f4a912c49d57cdafc40, State: Initialized, Role: FOLLOWER
I20260806 10:54:17.693979 7416 consensus_queue.cc:260] T 70d908fd179f40af99986c9315c8035e P ce7b4388b4584f4a912c49d57cdafc40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:17.694726 7416 ts_tablet_manager.cc:1428] T 70d908fd179f40af99986c9315c8035e P ce7b4388b4584f4a912c49d57cdafc40: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 10:54:17.694535 7423 tablet_replica.cc:439] T 70d908fd179f40af99986c9315c8035e P ce7b4388b4584f4a912c49d57cdafc40: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } } }
I20260806 10:54:17.694940 7423 tablet_replica.cc:442] T 70d908fd179f40af99986c9315c8035e P ce7b4388b4584f4a912c49d57cdafc40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:54:17.840502 7422 raft_consensus.cc:482] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:54:17.840799 7422 raft_consensus.cc:504] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:17.842736 7422 leader_election.cc:288] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee5505bb889f40dd989938bda8954e15 (127.30.218.1:38295), ce7b4388b4584f4a912c49d57cdafc40 (127.30.218.2:39917)
I20260806 10:54:17.866482 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:17.843266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:53291 (local address 127.30.218.2:39917)
0806 10:54:17.858000 (+ 14734us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:17.858009 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:17.858913 (+ 904us) server_negotiation.cc:407] Connection header received
0806 10:54:17.859034 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:17.859039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:17.859129 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:17.859254 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:17.859926 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.861046 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.864079 (+ 3033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.865413 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.865638 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:17.865728 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:17.865965 (+ 237us) server_negotiation.cc:299] Negotiation successful
0806 10:54:17.866148 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14533,"thread_start_us":15067,"threads_started":1}
I20260806 10:54:17.866569 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:17.843047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:39917 (local address 127.30.218.3:53291)
0806 10:54:17.858800 (+ 15753us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:17.858817 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:17.858921 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:17.859277 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:17.859283 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:17.859321 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:17.859758 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:17.859773 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.861232 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.861238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:17.863789 (+ 2551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:17.863813 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.865625 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.865631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:17.865785 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:17.865817 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 10:54:17.865911 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 10:54:17.866086 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15645,"mutex_wait_us":249,"spinlock_wait_cycles":1408,"thread_start_us":15367,"threads_started":1}
I20260806 10:54:17.866811 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:17.843291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:45429 (local address 127.30.218.1:38295)
0806 10:54:17.858462 (+ 15171us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:17.858470 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:17.858741 (+ 271us) server_negotiation.cc:407] Connection header received
0806 10:54:17.859011 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:17.859018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:17.859119 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:17.859254 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:17.859926 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.861093 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.864072 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.865413 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.865638 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:17.865728 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:17.866325 (+ 597us) server_negotiation.cc:299] Negotiation successful
0806 10:54:17.866490 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15013,"thread_start_us":14876,"threads_started":1}
I20260806 10:54:17.867228 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:17.843041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:38295 (local address 127.30.218.3:45429)
0806 10:54:17.858571 (+ 15530us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:17.858600 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:17.858826 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:17.859278 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:17.859283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:17.859330 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:17.859758 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:17.859773 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.861236 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.861241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:17.863789 (+ 2548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:17.863813 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.865910 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.865917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:17.866127 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:17.866161 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 10:54:17.866266 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 10:54:17.866404 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15368,"mutex_wait_us":100,"spinlock_wait_cycles":2944,"thread_start_us":15694,"threads_started":1}
I20260806 10:54:17.867648 7196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d908fd179f40af99986c9315c8035e" candidate_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce7b4388b4584f4a912c49d57cdafc40" is_pre_election: true
I20260806 10:54:17.867990 7053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d908fd179f40af99986c9315c8035e" candidate_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5505bb889f40dd989938bda8954e15" is_pre_election: true
I20260806 10:54:17.868090 7196 raft_consensus.cc:2454] T 70d908fd179f40af99986c9315c8035e P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3977d8fd8724b35b3d3cf52c96534bc in term 0.
I20260806 10:54:17.868321 7053 raft_consensus.cc:2454] T 70d908fd179f40af99986c9315c8035e P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3977d8fd8724b35b3d3cf52c96534bc in term 0.
I20260806 10:54:17.868769 7271 leader_election.cc:302] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: ce7b4388b4584f4a912c49d57cdafc40, ee5505bb889f40dd989938bda8954e15, f3977d8fd8724b35b3d3cf52c96534bc; no voters:
I20260806 10:54:17.869166 7422 raft_consensus.cc:2781] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:54:17.869283 7422 raft_consensus.cc:482] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:54:17.869375 7422 raft_consensus.cc:3037] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:17.872051 7422 raft_consensus.cc:504] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:17.872648 7422 leader_election.cc:288] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc [CANDIDATE]: Term 1 election: Requested vote from peers ee5505bb889f40dd989938bda8954e15 (127.30.218.1:38295), ce7b4388b4584f4a912c49d57cdafc40 (127.30.218.2:39917)
I20260806 10:54:17.873210 7053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d908fd179f40af99986c9315c8035e" candidate_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5505bb889f40dd989938bda8954e15"
I20260806 10:54:17.873210 7196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d908fd179f40af99986c9315c8035e" candidate_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce7b4388b4584f4a912c49d57cdafc40"
I20260806 10:54:17.873459 7196 raft_consensus.cc:3037] T 70d908fd179f40af99986c9315c8035e P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:17.873486 7053 raft_consensus.cc:3037] T 70d908fd179f40af99986c9315c8035e P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:17.876677 7196 raft_consensus.cc:2454] T 70d908fd179f40af99986c9315c8035e P ce7b4388b4584f4a912c49d57cdafc40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3977d8fd8724b35b3d3cf52c96534bc in term 1.
I20260806 10:54:17.876663 7053 raft_consensus.cc:2454] T 70d908fd179f40af99986c9315c8035e P ee5505bb889f40dd989938bda8954e15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3977d8fd8724b35b3d3cf52c96534bc in term 1.
I20260806 10:54:17.877224 7270 leader_election.cc:302] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee5505bb889f40dd989938bda8954e15, f3977d8fd8724b35b3d3cf52c96534bc; no voters:
I20260806 10:54:17.877560 7422 raft_consensus.cc:2781] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:54:17.877942 7422 raft_consensus.cc:686] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc [term 1 LEADER]: Becoming Leader. State: Replica: f3977d8fd8724b35b3d3cf52c96534bc, State: Running, Role: LEADER
I20260806 10:54:17.878342 7422 consensus_queue.cc:237] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:17.880187 7461 tablet_replica.cc:439] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc: TxnStatusTablet state changed. Reason: New leader f3977d8fd8724b35b3d3cf52c96534bc. Latest consensus state: current_term: 1 leader_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } } }
I20260806 10:54:17.880491 7461 tablet_replica.cc:442] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc: This TxnStatusTablet replica's current role is: LEADER
I20260806 10:54:17.881168 7463 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 10:54:17.881494 7463 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 10:54:17.882310 6894 catalog_manager.cc:5495] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc reported cstate change: term changed from 0 to 1, leader changed from <none> to f3977d8fd8724b35b3d3cf52c96534bc (127.30.218.3). New cstate: current_term: 1 leader_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } health_report { overall_health: HEALTHY } } }
I20260806 10:54:17.934125 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:17.926103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:45881 (local address 127.0.0.1:46922)
0806 10:54:17.926399 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:17.926422 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:17.926623 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:17.927372 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:17.927375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:17.927387 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 10:54:17.927578 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:17.927587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.929121 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.929126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:17.930584 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:17.930603 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.931727 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.931734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:17.931962 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:17.932059 (+ 97us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 10:54:17.933419 (+ 1360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 10:54:17.933480 (+ 61us) client_negotiation.cc:769] Sending connection context
0806 10:54:17.933573 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 10:54:17.933773 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 10:54:17.934150 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:17.926192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46922 (local address 127.30.218.3:45881)
0806 10:54:17.926889 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:17.926896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:17.926922 (+ 26us) server_negotiation.cc:407] Connection header received
0806 10:54:17.927039 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:17.927045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:17.927168 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:17.927365 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:54:17.927718 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.928979 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.930784 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:17.931519 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:17.931675 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:17.932343 (+ 668us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:54:17.933287 (+ 944us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:54:17.933485 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:17.933610 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 10:54:17.933714 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":382,"threads_started":1}
I20260806 10:54:17.945345 7196 raft_consensus.cc:1261] T 70d908fd179f40af99986c9315c8035e P ce7b4388b4584f4a912c49d57cdafc40 [term 1 FOLLOWER]: Refusing update from remote peer f3977d8fd8724b35b3d3cf52c96534bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:54:17.947062 7053 raft_consensus.cc:1261] T 70d908fd179f40af99986c9315c8035e P ee5505bb889f40dd989938bda8954e15 [term 1 FOLLOWER]: Refusing update from remote peer f3977d8fd8724b35b3d3cf52c96534bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:54:17.946955 7461 consensus_queue.cc:1035] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:54:17.948135 7461 consensus_queue.cc:1035] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:54:17.964147 7423 tablet_replica.cc:439] T 70d908fd179f40af99986c9315c8035e P ce7b4388b4584f4a912c49d57cdafc40: TxnStatusTablet state changed. Reason: New leader f3977d8fd8724b35b3d3cf52c96534bc. Latest consensus state: current_term: 1 leader_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } } }
I20260806 10:54:17.964540 7423 tablet_replica.cc:442] T 70d908fd179f40af99986c9315c8035e P ce7b4388b4584f4a912c49d57cdafc40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:54:17.967274 7467 mvcc.cc:205] Tried to move back new op lower bound from 7315511942931644416 to 7315511942673362944. Current Snapshot: MvccSnapshot[applied={T|T < 7315511942931644416}]
I20260806 10:54:17.967738 7468 mvcc.cc:205] Tried to move back new op lower bound from 7315511942931644416 to 7315511942673362944. Current Snapshot: MvccSnapshot[applied={T|T < 7315511942931644416}]
I20260806 10:54:17.968307 7421 tablet_replica.cc:439] T 70d908fd179f40af99986c9315c8035e P ee5505bb889f40dd989938bda8954e15: TxnStatusTablet state changed. Reason: New leader f3977d8fd8724b35b3d3cf52c96534bc. Latest consensus state: current_term: 1 leader_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } } }
I20260806 10:54:17.968674 7421 tablet_replica.cc:442] T 70d908fd179f40af99986c9315c8035e P ee5505bb889f40dd989938bda8954e15: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:54:17.978296 7461 tablet_replica.cc:439] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } } }
I20260806 10:54:17.978668 7461 tablet_replica.cc:442] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc: This TxnStatusTablet replica's current role is: LEADER
I20260806 10:54:17.979168 7461 tablet_replica.cc:439] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } } }
I20260806 10:54:17.979395 7461 tablet_replica.cc:442] T 70d908fd179f40af99986c9315c8035e P f3977d8fd8724b35b3d3cf52c96534bc: This TxnStatusTablet replica's current role is: LEADER
I20260806 10:54:17.980037 7421 tablet_replica.cc:439] T 70d908fd179f40af99986c9315c8035e P ee5505bb889f40dd989938bda8954e15: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } } }
I20260806 10:54:17.980453 7421 tablet_replica.cc:442] T 70d908fd179f40af99986c9315c8035e P ee5505bb889f40dd989938bda8954e15: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:54:17.983217 7423 tablet_replica.cc:439] T 70d908fd179f40af99986c9315c8035e P ce7b4388b4584f4a912c49d57cdafc40: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } } }
I20260806 10:54:17.984212 7423 tablet_replica.cc:442] T 70d908fd179f40af99986c9315c8035e P ce7b4388b4584f4a912c49d57cdafc40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 10:54:18.005828 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:17.999402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:45881 (local address 127.30.218.2:55447)
0806 10:54:18.000083 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:18.000099 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:18.000250 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:18.000609 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:18.000612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:18.000622 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 10:54:18.000907 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:18.000916 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.002399 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.002402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:18.003478 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:18.003493 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.004545 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.004550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:18.004686 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:18.004810 (+ 124us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 10:54:18.005248 (+ 438us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 10:54:18.005257 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 10:54:18.005323 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 10:54:18.005483 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":366,"threads_started":1}
I20260806 10:54:18.005829 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:17.999408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:55447 (local address 127.30.218.3:45881)
0806 10:54:17.999575 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:17.999579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:18.000204 (+ 625us) server_negotiation.cc:407] Connection header received
0806 10:54:18.000384 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:18.000393 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:18.000491 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:18.000604 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:54:18.001052 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.002262 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.003658 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.004350 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.004494 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:18.004993 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:54:18.005146 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:54:18.005240 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:18.005372 (+ 132us) server_negotiation.cc:299] Negotiation successful
0806 10:54:18.005492 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 10:54:18.014791 7423 consensus_queue.cc:1035] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:54:18.020195 7053 raft_consensus.cc:1261] T 11635649ca354ef18e9df20cca74d307 P ee5505bb889f40dd989938bda8954e15 [term 1 FOLLOWER]: Refusing update from remote peer ce7b4388b4584f4a912c49d57cdafc40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:54:18.020902 7423 consensus_queue.cc:1035] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:54:18.064386 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:18.056719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:56431 (local address 127.30.218.2:39917)
0806 10:54:18.056938 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:18.056945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:18.057675 (+ 730us) server_negotiation.cc:407] Connection header received
0806 10:54:18.057929 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:18.057935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:18.058009 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:18.058086 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:54:18.059293 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.060069 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.061853 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.062401 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.062523 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:18.063489 (+ 966us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:54:18.063672 (+ 183us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:54:18.063772 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:18.063921 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 10:54:18.064052 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 10:54:18.064474 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:18.056706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:39917 (local address 127.30.218.3:56431)
0806 10:54:18.057539 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:18.057562 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:18.057761 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:18.058476 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:18.058480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:18.058492 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 10:54:18.058972 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:18.058989 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.060222 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.060228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:18.061692 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:18.061708 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.062976 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.062982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:18.063157 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:18.063252 (+ 95us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 10:54:18.063780 (+ 528us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 10:54:18.063795 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 10:54:18.063883 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 10:54:18.064043 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":404,"threads_started":1}
I20260806 10:54:18.113857 6894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:54:18.114219 6894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:54:18.118242 6894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9: Sending DeleteTablet for 3 replicas of tablet 11635649ca354ef18e9df20cca74d307
I20260806 10:54:18.119473 7032 tablet_service.cc:1514] Processing DeleteTablet for tablet 11635649ca354ef18e9df20cca74d307 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:54:18 UTC) from {username='oryx', principal='oryx/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:48114
I20260806 10:54:18.119516 7318 tablet_service.cc:1514] Processing DeleteTablet for tablet 11635649ca354ef18e9df20cca74d307 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:54:18 UTC) from {username='oryx', principal='oryx/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:46906
I20260806 10:54:18.119611 7175 tablet_service.cc:1514] Processing DeleteTablet for tablet 11635649ca354ef18e9df20cca74d307 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:54:18 UTC) from {username='oryx', principal='oryx/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:43534
I20260806 10:54:18.131882 7487 tablet_replica.cc:330] T 11635649ca354ef18e9df20cca74d307 P f3977d8fd8724b35b3d3cf52c96534bc: stopping tablet replica
I20260806 10:54:18.132198 7487 raft_consensus.cc:2229] T 11635649ca354ef18e9df20cca74d307 P f3977d8fd8724b35b3d3cf52c96534bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:54:18.132529 7487 raft_consensus.cc:2258] T 11635649ca354ef18e9df20cca74d307 P f3977d8fd8724b35b3d3cf52c96534bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:54:18.136179 7486 tablet_replica.cc:330] T 11635649ca354ef18e9df20cca74d307 P ee5505bb889f40dd989938bda8954e15: stopping tablet replica
I20260806 10:54:18.136430 7486 raft_consensus.cc:2229] T 11635649ca354ef18e9df20cca74d307 P ee5505bb889f40dd989938bda8954e15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:54:18.136726 7486 raft_consensus.cc:2258] T 11635649ca354ef18e9df20cca74d307 P ee5505bb889f40dd989938bda8954e15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:54:18.138163 7486 ts_tablet_manager.cc:1905] T 11635649ca354ef18e9df20cca74d307 P ee5505bb889f40dd989938bda8954e15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:54:18.143441 7487 ts_tablet_manager.cc:1905] T 11635649ca354ef18e9df20cca74d307 P f3977d8fd8724b35b3d3cf52c96534bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:54:18.144439 7488 tablet_replica.cc:330] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40: stopping tablet replica
I20260806 10:54:18.144840 7488 raft_consensus.cc:2229] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:54:18.146126 7486 ts_tablet_manager.cc:1918] T 11635649ca354ef18e9df20cca74d307 P ee5505bb889f40dd989938bda8954e15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:54:18.146293 7486 log.cc:1196] T 11635649ca354ef18e9df20cca74d307 P ee5505bb889f40dd989938bda8954e15: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-0/wal/wals/11635649ca354ef18e9df20cca74d307
I20260806 10:54:18.146803 7486 ts_tablet_manager.cc:1939] T 11635649ca354ef18e9df20cca74d307 P ee5505bb889f40dd989938bda8954e15: Deleting consensus metadata
I20260806 10:54:18.146822 7488 raft_consensus.cc:2258] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:54:18.148130 6880 catalog_manager.cc:4841] TS ee5505bb889f40dd989938bda8954e15 (127.30.218.1:38295): tablet 11635649ca354ef18e9df20cca74d307 (table test-table [id=42df7ee50db345098828214153b0e182]) successfully deleted
I20260806 10:54:18.149010 7488 ts_tablet_manager.cc:1905] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:54:18.156805 7487 ts_tablet_manager.cc:1918] T 11635649ca354ef18e9df20cca74d307 P f3977d8fd8724b35b3d3cf52c96534bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:54:18.156994 7487 log.cc:1196] T 11635649ca354ef18e9df20cca74d307 P f3977d8fd8724b35b3d3cf52c96534bc: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/11635649ca354ef18e9df20cca74d307
I20260806 10:54:18.157518 7487 ts_tablet_manager.cc:1939] T 11635649ca354ef18e9df20cca74d307 P f3977d8fd8724b35b3d3cf52c96534bc: Deleting consensus metadata
I20260806 10:54:18.158759 7488 ts_tablet_manager.cc:1918] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:54:18.158916 7488 log.cc:1196] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/11635649ca354ef18e9df20cca74d307
I20260806 10:54:18.158916 6878 catalog_manager.cc:4841] TS f3977d8fd8724b35b3d3cf52c96534bc (127.30.218.3:45881): tablet 11635649ca354ef18e9df20cca74d307 (table test-table [id=42df7ee50db345098828214153b0e182]) successfully deleted
I20260806 10:54:18.159404 7488 ts_tablet_manager.cc:1939] T 11635649ca354ef18e9df20cca74d307 P ce7b4388b4584f4a912c49d57cdafc40: Deleting consensus metadata
I20260806 10:54:18.160725 6881 catalog_manager.cc:4841] TS ce7b4388b4584f4a912c49d57cdafc40 (127.30.218.2:39917): tablet 11635649ca354ef18e9df20cca74d307 (table test-table [id=42df7ee50db345098828214153b0e182]) successfully deleted
Aug 06 10:54:18 dist-test-slave-49h3 krb5kdc[6848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 10:54:18 dist-test-slave-49h3 krb5kdc[6848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:18 dist-test-slave-49h3 krb5kdc[6848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 10:54:18.221980 7496 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:18.208457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58826 (local address 127.30.218.62:42985)
0806 10:54:18.209021 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:18.209029 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:18.209343 (+ 314us) server_negotiation.cc:407] Connection header received
0806 10:54:18.209535 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:18.209540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:18.209638 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:18.209777 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:18.211377 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.212605 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.214103 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.214770 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.214928 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:18.221231 (+ 6303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58826: BlockingRecv error: recv got EOF from 127.0.0.1:58826 (error 108)
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":274,"threads_started":1}
Aug 06 10:54:18 dist-test-slave-49h3 krb5kdc[6848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM
I20260806 10:54:18.243126 7496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:18.224506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58842 (local address 127.30.218.62:42985)
0806 10:54:18.224745 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:18.224752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:18.226007 (+ 1255us) server_negotiation.cc:407] Connection header received
0806 10:54:18.226180 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:18.226186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:18.226295 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:18.226424 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:18.227626 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.228810 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.230191 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.230873 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.231028 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:18.235576 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:18.235586 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:18.235592 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:18.235650 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:18.239678 (+ 4028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:18.240642 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:18.240649 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:18.240652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:18.240871 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:18.241362 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:18.241368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:18.241371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:18.241837 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:18.242160 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:18.242471 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 10:54:18.242750 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 10:54:18.247709 6894 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:54:18.248572 6894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:54:18.256733 7032 tablet_service.cc:1467] Processing CreateTablet for tablet 8d6177154ba34152acc7e092cc2d69c4 (DEFAULT_TABLE table=test-table [id=2f991d27db034379a8fb2baad586af47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:54:18.257339 7032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d6177154ba34152acc7e092cc2d69c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:18.257558 7175 tablet_service.cc:1467] Processing CreateTablet for tablet 8d6177154ba34152acc7e092cc2d69c4 (DEFAULT_TABLE table=test-table [id=2f991d27db034379a8fb2baad586af47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:54:18.257612 7318 tablet_service.cc:1467] Processing CreateTablet for tablet 8d6177154ba34152acc7e092cc2d69c4 (DEFAULT_TABLE table=test-table [id=2f991d27db034379a8fb2baad586af47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:54:18.258033 7175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d6177154ba34152acc7e092cc2d69c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:18.258112 7318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d6177154ba34152acc7e092cc2d69c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:18.271265 7502 tablet_bootstrap.cc:492] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15: Bootstrap starting.
I20260806 10:54:18.272320 7503 tablet_bootstrap.cc:492] T 8d6177154ba34152acc7e092cc2d69c4 P f3977d8fd8724b35b3d3cf52c96534bc: Bootstrap starting.
I20260806 10:54:18.272703 7504 tablet_bootstrap.cc:492] T 8d6177154ba34152acc7e092cc2d69c4 P ce7b4388b4584f4a912c49d57cdafc40: Bootstrap starting.
I20260806 10:54:18.274959 7503 tablet_bootstrap.cc:654] T 8d6177154ba34152acc7e092cc2d69c4 P f3977d8fd8724b35b3d3cf52c96534bc: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:18.275374 7502 tablet_bootstrap.cc:654] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:18.276368 7504 tablet_bootstrap.cc:654] T 8d6177154ba34152acc7e092cc2d69c4 P ce7b4388b4584f4a912c49d57cdafc40: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:18.284389 7502 tablet_bootstrap.cc:492] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15: No bootstrap required, opened a new log
I20260806 10:54:18.284410 7503 tablet_bootstrap.cc:492] T 8d6177154ba34152acc7e092cc2d69c4 P f3977d8fd8724b35b3d3cf52c96534bc: No bootstrap required, opened a new log
I20260806 10:54:18.284596 7502 ts_tablet_manager.cc:1397] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260806 10:54:18.284686 7503 ts_tablet_manager.cc:1397] T 8d6177154ba34152acc7e092cc2d69c4 P f3977d8fd8724b35b3d3cf52c96534bc: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.011s
I20260806 10:54:18.285616 7503 raft_consensus.cc:348] T 8d6177154ba34152acc7e092cc2d69c4 P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:18.285579 7502 raft_consensus.cc:348] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:18.285889 7502 raft_consensus.cc:374] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:18.285897 7503 raft_consensus.cc:374] T 8d6177154ba34152acc7e092cc2d69c4 P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:18.285989 7502 raft_consensus.cc:729] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee5505bb889f40dd989938bda8954e15, State: Initialized, Role: FOLLOWER
I20260806 10:54:18.286006 7503 raft_consensus.cc:729] T 8d6177154ba34152acc7e092cc2d69c4 P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3977d8fd8724b35b3d3cf52c96534bc, State: Initialized, Role: FOLLOWER
I20260806 10:54:18.286334 7502 consensus_queue.cc:260] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:18.286324 7503 consensus_queue.cc:260] T 8d6177154ba34152acc7e092cc2d69c4 P f3977d8fd8724b35b3d3cf52c96534bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:18.287729 7503 ts_tablet_manager.cc:1428] T 8d6177154ba34152acc7e092cc2d69c4 P f3977d8fd8724b35b3d3cf52c96534bc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 10:54:18.289904 7502 ts_tablet_manager.cc:1428] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260806 10:54:18.291355 7504 tablet_bootstrap.cc:492] T 8d6177154ba34152acc7e092cc2d69c4 P ce7b4388b4584f4a912c49d57cdafc40: No bootstrap required, opened a new log
I20260806 10:54:18.291641 7504 ts_tablet_manager.cc:1397] T 8d6177154ba34152acc7e092cc2d69c4 P ce7b4388b4584f4a912c49d57cdafc40: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.004s
I20260806 10:54:18.292685 7504 raft_consensus.cc:348] T 8d6177154ba34152acc7e092cc2d69c4 P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:18.293006 7504 raft_consensus.cc:374] T 8d6177154ba34152acc7e092cc2d69c4 P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:18.293126 7504 raft_consensus.cc:729] T 8d6177154ba34152acc7e092cc2d69c4 P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce7b4388b4584f4a912c49d57cdafc40, State: Initialized, Role: FOLLOWER
I20260806 10:54:18.293413 7504 consensus_queue.cc:260] T 8d6177154ba34152acc7e092cc2d69c4 P ce7b4388b4584f4a912c49d57cdafc40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:18.294116 7504 ts_tablet_manager.cc:1428] T 8d6177154ba34152acc7e092cc2d69c4 P ce7b4388b4584f4a912c49d57cdafc40: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 10:54:18.393250 7421 raft_consensus.cc:482] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:54:18.393589 7421 raft_consensus.cc:504] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:18.395416 7421 leader_election.cc:288] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce7b4388b4584f4a912c49d57cdafc40 (127.30.218.2:39917), f3977d8fd8724b35b3d3cf52c96534bc (127.30.218.3:45881)
I20260806 10:54:18.402357 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:18.395827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:43707 (local address 127.30.218.2:39917)
0806 10:54:18.396029 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:18.396036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:18.396623 (+ 587us) server_negotiation.cc:407] Connection header received
0806 10:54:18.396912 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:18.396919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:18.397002 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:18.397101 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:18.397845 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.398649 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.400478 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.401319 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.401535 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:18.401616 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:18.402006 (+ 390us) server_negotiation.cc:299] Negotiation successful
0806 10:54:18.402138 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 10:54:18.402406 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:18.396223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:44851 (local address 127.30.218.3:45881)
0806 10:54:18.396416 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:18.396422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:18.396939 (+ 517us) server_negotiation.cc:407] Connection header received
0806 10:54:18.397098 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:18.397104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:18.397179 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:18.397275 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:18.397977 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.398664 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.400518 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.401326 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.401522 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:18.401614 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:18.401983 (+ 369us) server_negotiation.cc:299] Negotiation successful
0806 10:54:18.402132 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 10:54:18.402509 7508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:18.395746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:45881 (local address 127.30.218.1:44851)
0806 10:54:18.396824 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:18.396843 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:18.396970 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:18.397313 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:18.397318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:18.397345 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:18.397812 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:18.397823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.398817 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.398821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:18.400344 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:18.400358 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.401575 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.401582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:18.401785 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:18.401839 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 10:54:18.401931 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 10:54:18.402088 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"mutex_wait_us":29,"thread_start_us":454,"threads_started":1}
I20260806 10:54:18.403103 7509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:18.395603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:39917 (local address 127.30.218.1:43707)
0806 10:54:18.396483 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:18.396506 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:18.396746 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:18.397293 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:18.397297 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:18.397321 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:18.397677 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:18.397689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.398792 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.398797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:18.400298 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:18.400313 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.401532 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.401539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:18.401783 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:18.401839 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 10:54:18.402601 (+ 762us) client_negotiation.cc:240] Negotiation successful
0806 10:54:18.402755 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":44,"thread_start_us":943,"threads_started":1}
I20260806 10:54:18.403146 7337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6177154ba34152acc7e092cc2d69c4" candidate_uuid: "ee5505bb889f40dd989938bda8954e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" is_pre_election: true
I20260806 10:54:18.403445 7337 raft_consensus.cc:2454] T 8d6177154ba34152acc7e092cc2d69c4 P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee5505bb889f40dd989938bda8954e15 in term 0.
I20260806 10:54:18.403780 7196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6177154ba34152acc7e092cc2d69c4" candidate_uuid: "ee5505bb889f40dd989938bda8954e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce7b4388b4584f4a912c49d57cdafc40" is_pre_election: true
I20260806 10:54:18.404060 7196 raft_consensus.cc:2454] T 8d6177154ba34152acc7e092cc2d69c4 P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee5505bb889f40dd989938bda8954e15 in term 0.
I20260806 10:54:18.403975 6983 leader_election.cc:302] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee5505bb889f40dd989938bda8954e15, f3977d8fd8724b35b3d3cf52c96534bc; no voters:
I20260806 10:54:18.404459 7421 raft_consensus.cc:2781] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:54:18.404580 7421 raft_consensus.cc:482] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:54:18.404681 7421 raft_consensus.cc:3037] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:18.407073 7421 raft_consensus.cc:504] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:18.407717 7421 leader_election.cc:288] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15 [CANDIDATE]: Term 1 election: Requested vote from peers ce7b4388b4584f4a912c49d57cdafc40 (127.30.218.2:39917), f3977d8fd8724b35b3d3cf52c96534bc (127.30.218.3:45881)
I20260806 10:54:18.408391 7196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6177154ba34152acc7e092cc2d69c4" candidate_uuid: "ee5505bb889f40dd989938bda8954e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce7b4388b4584f4a912c49d57cdafc40"
I20260806 10:54:18.408391 7337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6177154ba34152acc7e092cc2d69c4" candidate_uuid: "ee5505bb889f40dd989938bda8954e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3977d8fd8724b35b3d3cf52c96534bc"
I20260806 10:54:18.408629 7337 raft_consensus.cc:3037] T 8d6177154ba34152acc7e092cc2d69c4 P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:18.408629 7196 raft_consensus.cc:3037] T 8d6177154ba34152acc7e092cc2d69c4 P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:18.411821 7196 raft_consensus.cc:2454] T 8d6177154ba34152acc7e092cc2d69c4 P ce7b4388b4584f4a912c49d57cdafc40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee5505bb889f40dd989938bda8954e15 in term 1.
I20260806 10:54:18.411821 7337 raft_consensus.cc:2454] T 8d6177154ba34152acc7e092cc2d69c4 P f3977d8fd8724b35b3d3cf52c96534bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee5505bb889f40dd989938bda8954e15 in term 1.
I20260806 10:54:18.412438 6986 leader_election.cc:302] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: ce7b4388b4584f4a912c49d57cdafc40, ee5505bb889f40dd989938bda8954e15, f3977d8fd8724b35b3d3cf52c96534bc; no voters:
I20260806 10:54:18.412823 7421 raft_consensus.cc:2781] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:54:18.413296 7421 raft_consensus.cc:686] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15 [term 1 LEADER]: Becoming Leader. State: Replica: ee5505bb889f40dd989938bda8954e15, State: Running, Role: LEADER
I20260806 10:54:18.413779 7421 consensus_queue.cc:237] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:18.417469 6894 catalog_manager.cc:5495] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee5505bb889f40dd989938bda8954e15 (127.30.218.1). New cstate: current_term: 1 leader_uuid: "ee5505bb889f40dd989938bda8954e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } health_report { overall_health: UNKNOWN } } }
I20260806 10:54:18.435895 7513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:18.429568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48116 (local address 127.30.218.1:38295)
0806 10:54:18.430171 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:18.430176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:18.430196 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:54:18.430292 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:18.430296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:18.430376 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:18.430536 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:54:18.430973 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.431775 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.433141 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.433691 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.433810 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:18.434343 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:54:18.435121 (+ 778us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:54:18.435290 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:18.435447 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 10:54:18.435543 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":325,"threads_started":1}
I20260806 10:54:18.439786 7337 raft_consensus.cc:1261] T 8d6177154ba34152acc7e092cc2d69c4 P f3977d8fd8724b35b3d3cf52c96534bc [term 1 FOLLOWER]: Refusing update from remote peer ee5505bb889f40dd989938bda8954e15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:54:18.439855 7196 raft_consensus.cc:1261] T 8d6177154ba34152acc7e092cc2d69c4 P ce7b4388b4584f4a912c49d57cdafc40 [term 1 FOLLOWER]: Refusing update from remote peer ee5505bb889f40dd989938bda8954e15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:54:18.440630 7421 consensus_queue.cc:1035] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:54:18.440955 7510 consensus_queue.cc:1035] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:54:18.458113 6894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:54:18.458375 6894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:54:18.461585 6894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9: Sending DeleteTablet for 3 replicas of tablet 8d6177154ba34152acc7e092cc2d69c4
I20260806 10:54:18.462527 7175 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d6177154ba34152acc7e092cc2d69c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:54:18 UTC) from {username='oryx', principal='oryx/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:43534
I20260806 10:54:18.462540 7032 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d6177154ba34152acc7e092cc2d69c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:54:18 UTC) from {username='oryx', principal='oryx/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:48114
I20260806 10:54:18.462731 7318 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d6177154ba34152acc7e092cc2d69c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:54:18 UTC) from {username='oryx', principal='oryx/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:46906
I20260806 10:54:18.462864 7488 tablet_replica.cc:330] T 8d6177154ba34152acc7e092cc2d69c4 P ce7b4388b4584f4a912c49d57cdafc40: stopping tablet replica
I20260806 10:54:18.462906 7486 tablet_replica.cc:330] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15: stopping tablet replica
I20260806 10:54:18.463069 7488 raft_consensus.cc:2229] T 8d6177154ba34152acc7e092cc2d69c4 P ce7b4388b4584f4a912c49d57cdafc40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:54:18.463097 7486 raft_consensus.cc:2229] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:54:18.463366 7488 raft_consensus.cc:2258] T 8d6177154ba34152acc7e092cc2d69c4 P ce7b4388b4584f4a912c49d57cdafc40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:54:18.463690 7487 tablet_replica.cc:330] T 8d6177154ba34152acc7e092cc2d69c4 P f3977d8fd8724b35b3d3cf52c96534bc: stopping tablet replica
I20260806 10:54:18.463663 7486 raft_consensus.cc:2258] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:54:18.463914 7487 raft_consensus.cc:2229] T 8d6177154ba34152acc7e092cc2d69c4 P f3977d8fd8724b35b3d3cf52c96534bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:54:18.464118 7487 raft_consensus.cc:2258] T 8d6177154ba34152acc7e092cc2d69c4 P f3977d8fd8724b35b3d3cf52c96534bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:54:18.465024 7486 ts_tablet_manager.cc:1905] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:54:18.464877 6893 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:54:18.465639 6893 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:54:18.465780 7487 ts_tablet_manager.cc:1905] T 8d6177154ba34152acc7e092cc2d69c4 P f3977d8fd8724b35b3d3cf52c96534bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:54:18.466399 7488 ts_tablet_manager.cc:1905] T 8d6177154ba34152acc7e092cc2d69c4 P ce7b4388b4584f4a912c49d57cdafc40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:54:18.473378 7488 ts_tablet_manager.cc:1918] T 8d6177154ba34152acc7e092cc2d69c4 P ce7b4388b4584f4a912c49d57cdafc40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:54:18.473598 7488 log.cc:1196] T 8d6177154ba34152acc7e092cc2d69c4 P ce7b4388b4584f4a912c49d57cdafc40: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/8d6177154ba34152acc7e092cc2d69c4
I20260806 10:54:18.474006 7488 ts_tablet_manager.cc:1939] T 8d6177154ba34152acc7e092cc2d69c4 P ce7b4388b4584f4a912c49d57cdafc40: Deleting consensus metadata
I20260806 10:54:18.474165 7487 ts_tablet_manager.cc:1918] T 8d6177154ba34152acc7e092cc2d69c4 P f3977d8fd8724b35b3d3cf52c96534bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:54:18.474699 7175 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5fb7eeb3944688998fbc6f3625afbc (DEFAULT_TABLE table=test-table [id=f0213beb51204686afd7ed53c4770fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:54:18.474958 7487 log.cc:1196] T 8d6177154ba34152acc7e092cc2d69c4 P f3977d8fd8724b35b3d3cf52c96534bc: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/8d6177154ba34152acc7e092cc2d69c4
I20260806 10:54:18.475201 7175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e5fb7eeb3944688998fbc6f3625afbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:18.475492 7487 ts_tablet_manager.cc:1939] T 8d6177154ba34152acc7e092cc2d69c4 P f3977d8fd8724b35b3d3cf52c96534bc: Deleting consensus metadata
I20260806 10:54:18.477692 6881 catalog_manager.cc:4841] TS ce7b4388b4584f4a912c49d57cdafc40 (127.30.218.2:39917): tablet 8d6177154ba34152acc7e092cc2d69c4 (table test-table [id=2f991d27db034379a8fb2baad586af47]) successfully deleted
I20260806 10:54:18.478277 6878 catalog_manager.cc:4841] TS f3977d8fd8724b35b3d3cf52c96534bc (127.30.218.3:45881): tablet 8d6177154ba34152acc7e092cc2d69c4 (table test-table [id=2f991d27db034379a8fb2baad586af47]) successfully deleted
I20260806 10:54:18.478821 7032 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5fb7eeb3944688998fbc6f3625afbc (DEFAULT_TABLE table=test-table [id=f0213beb51204686afd7ed53c4770fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:54:18.479040 7318 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5fb7eeb3944688998fbc6f3625afbc (DEFAULT_TABLE table=test-table [id=f0213beb51204686afd7ed53c4770fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:54:18.479337 7032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e5fb7eeb3944688998fbc6f3625afbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:18.479468 7318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e5fb7eeb3944688998fbc6f3625afbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:18.483817 7486 ts_tablet_manager.cc:1918] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:54:18.484046 7486 log.cc:1196] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-0/wal/wals/8d6177154ba34152acc7e092cc2d69c4
I20260806 10:54:18.484555 7486 ts_tablet_manager.cc:1939] T 8d6177154ba34152acc7e092cc2d69c4 P ee5505bb889f40dd989938bda8954e15: Deleting consensus metadata
I20260806 10:54:18.485936 6880 catalog_manager.cc:4841] TS ee5505bb889f40dd989938bda8954e15 (127.30.218.1:38295): tablet 8d6177154ba34152acc7e092cc2d69c4 (table test-table [id=2f991d27db034379a8fb2baad586af47]) successfully deleted
I20260806 10:54:18.486677 7502 tablet_bootstrap.cc:492] T 8e5fb7eeb3944688998fbc6f3625afbc P ee5505bb889f40dd989938bda8954e15: Bootstrap starting.
I20260806 10:54:18.488217 7502 tablet_bootstrap.cc:654] T 8e5fb7eeb3944688998fbc6f3625afbc P ee5505bb889f40dd989938bda8954e15: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:18.490705 7502 tablet_bootstrap.cc:492] T 8e5fb7eeb3944688998fbc6f3625afbc P ee5505bb889f40dd989938bda8954e15: No bootstrap required, opened a new log
I20260806 10:54:18.490772 7503 tablet_bootstrap.cc:492] T 8e5fb7eeb3944688998fbc6f3625afbc P f3977d8fd8724b35b3d3cf52c96534bc: Bootstrap starting.
I20260806 10:54:18.490883 7502 ts_tablet_manager.cc:1397] T 8e5fb7eeb3944688998fbc6f3625afbc P ee5505bb889f40dd989938bda8954e15: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260806 10:54:18.491195 7504 tablet_bootstrap.cc:492] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40: Bootstrap starting.
I20260806 10:54:18.491693 7502 raft_consensus.cc:348] T 8e5fb7eeb3944688998fbc6f3625afbc P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:18.491947 7502 raft_consensus.cc:374] T 8e5fb7eeb3944688998fbc6f3625afbc P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:18.492041 7502 raft_consensus.cc:729] T 8e5fb7eeb3944688998fbc6f3625afbc P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee5505bb889f40dd989938bda8954e15, State: Initialized, Role: FOLLOWER
I20260806 10:54:18.492435 7503 tablet_bootstrap.cc:654] T 8e5fb7eeb3944688998fbc6f3625afbc P f3977d8fd8724b35b3d3cf52c96534bc: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:18.492328 7502 consensus_queue.cc:260] T 8e5fb7eeb3944688998fbc6f3625afbc P ee5505bb889f40dd989938bda8954e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:18.492847 7504 tablet_bootstrap.cc:654] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:18.492884 7502 ts_tablet_manager.cc:1428] T 8e5fb7eeb3944688998fbc6f3625afbc P ee5505bb889f40dd989938bda8954e15: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 10:54:18.495131 7503 tablet_bootstrap.cc:492] T 8e5fb7eeb3944688998fbc6f3625afbc P f3977d8fd8724b35b3d3cf52c96534bc: No bootstrap required, opened a new log
I20260806 10:54:18.495675 7504 tablet_bootstrap.cc:492] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40: No bootstrap required, opened a new log
I20260806 10:54:18.495782 7503 ts_tablet_manager.cc:1397] T 8e5fb7eeb3944688998fbc6f3625afbc P f3977d8fd8724b35b3d3cf52c96534bc: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260806 10:54:18.495838 7504 ts_tablet_manager.cc:1397] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 10:54:18.496580 7504 raft_consensus.cc:348] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:18.496580 7503 raft_consensus.cc:348] T 8e5fb7eeb3944688998fbc6f3625afbc P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:18.496834 7504 raft_consensus.cc:374] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:18.496918 7504 raft_consensus.cc:729] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce7b4388b4584f4a912c49d57cdafc40, State: Initialized, Role: FOLLOWER
I20260806 10:54:18.496959 7503 raft_consensus.cc:374] T 8e5fb7eeb3944688998fbc6f3625afbc P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:18.497138 7503 raft_consensus.cc:729] T 8e5fb7eeb3944688998fbc6f3625afbc P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3977d8fd8724b35b3d3cf52c96534bc, State: Initialized, Role: FOLLOWER
I20260806 10:54:18.497193 7504 consensus_queue.cc:260] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:18.497845 7504 ts_tablet_manager.cc:1428] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 10:54:18.498148 7503 consensus_queue.cc:260] T 8e5fb7eeb3944688998fbc6f3625afbc P f3977d8fd8724b35b3d3cf52c96534bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:18.499316 7503 ts_tablet_manager.cc:1428] T 8e5fb7eeb3944688998fbc6f3625afbc P f3977d8fd8724b35b3d3cf52c96534bc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 10:54:18.569545 7482 raft_consensus.cc:482] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:54:18.569766 7482 raft_consensus.cc:504] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:18.570485 7482 leader_election.cc:288] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee5505bb889f40dd989938bda8954e15 (127.30.218.1:38295), f3977d8fd8724b35b3d3cf52c96534bc (127.30.218.3:45881)
I20260806 10:54:18.571118 7337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5fb7eeb3944688998fbc6f3625afbc" candidate_uuid: "ce7b4388b4584f4a912c49d57cdafc40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" is_pre_election: true
I20260806 10:54:18.571182 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5fb7eeb3944688998fbc6f3625afbc" candidate_uuid: "ce7b4388b4584f4a912c49d57cdafc40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5505bb889f40dd989938bda8954e15" is_pre_election: true
I20260806 10:54:18.571386 7337 raft_consensus.cc:2454] T 8e5fb7eeb3944688998fbc6f3625afbc P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce7b4388b4584f4a912c49d57cdafc40 in term 0.
I20260806 10:54:18.571547 7052 raft_consensus.cc:2454] T 8e5fb7eeb3944688998fbc6f3625afbc P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce7b4388b4584f4a912c49d57cdafc40 in term 0.
I20260806 10:54:18.572090 7127 leader_election.cc:302] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce7b4388b4584f4a912c49d57cdafc40, ee5505bb889f40dd989938bda8954e15; no voters:
I20260806 10:54:18.572382 7482 raft_consensus.cc:2781] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:54:18.572503 7482 raft_consensus.cc:482] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:54:18.572599 7482 raft_consensus.cc:3037] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:18.574942 7482 raft_consensus.cc:504] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:18.575587 7482 leader_election.cc:288] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40 [CANDIDATE]: Term 1 election: Requested vote from peers ee5505bb889f40dd989938bda8954e15 (127.30.218.1:38295), f3977d8fd8724b35b3d3cf52c96534bc (127.30.218.3:45881)
I20260806 10:54:18.576256 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5fb7eeb3944688998fbc6f3625afbc" candidate_uuid: "ce7b4388b4584f4a912c49d57cdafc40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5505bb889f40dd989938bda8954e15"
I20260806 10:54:18.576231 7337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5fb7eeb3944688998fbc6f3625afbc" candidate_uuid: "ce7b4388b4584f4a912c49d57cdafc40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3977d8fd8724b35b3d3cf52c96534bc"
I20260806 10:54:18.576471 7052 raft_consensus.cc:3037] T 8e5fb7eeb3944688998fbc6f3625afbc P ee5505bb889f40dd989938bda8954e15 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:18.576478 7337 raft_consensus.cc:3037] T 8e5fb7eeb3944688998fbc6f3625afbc P f3977d8fd8724b35b3d3cf52c96534bc [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:18.579321 7052 raft_consensus.cc:2454] T 8e5fb7eeb3944688998fbc6f3625afbc P ee5505bb889f40dd989938bda8954e15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce7b4388b4584f4a912c49d57cdafc40 in term 1.
I20260806 10:54:18.579461 7337 raft_consensus.cc:2454] T 8e5fb7eeb3944688998fbc6f3625afbc P f3977d8fd8724b35b3d3cf52c96534bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce7b4388b4584f4a912c49d57cdafc40 in term 1.
I20260806 10:54:18.579838 7127 leader_election.cc:302] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce7b4388b4584f4a912c49d57cdafc40, ee5505bb889f40dd989938bda8954e15; no voters:
I20260806 10:54:18.580199 7482 raft_consensus.cc:2781] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:54:18.580366 7482 raft_consensus.cc:686] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40 [term 1 LEADER]: Becoming Leader. State: Replica: ce7b4388b4584f4a912c49d57cdafc40, State: Running, Role: LEADER
I20260806 10:54:18.580672 7482 consensus_queue.cc:237] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } }
I20260806 10:54:18.583535 6894 catalog_manager.cc:5495] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce7b4388b4584f4a912c49d57cdafc40 (127.30.218.2). New cstate: current_term: 1 leader_uuid: "ce7b4388b4584f4a912c49d57cdafc40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce7b4388b4584f4a912c49d57cdafc40" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 } health_report { overall_health: UNKNOWN } } }
I20260806 10:54:18.620672 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:18.613504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43558 (local address 127.30.218.2:39917)
0806 10:54:18.613709 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:18.613717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:18.613743 (+ 26us) server_negotiation.cc:407] Connection header received
0806 10:54:18.613986 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:18.613992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:18.614124 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:18.614227 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:54:18.614705 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.615894 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.617977 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.618553 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.618689 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:18.619361 (+ 672us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:54:18.619506 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:54:18.619594 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:18.620209 (+ 615us) server_negotiation.cc:299] Negotiation successful
0806 10:54:18.620315 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 10:54:18.632426 7052 raft_consensus.cc:1261] T 8e5fb7eeb3944688998fbc6f3625afbc P ee5505bb889f40dd989938bda8954e15 [term 1 FOLLOWER]: Refusing update from remote peer ce7b4388b4584f4a912c49d57cdafc40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:54:18.632467 7337 raft_consensus.cc:1261] T 8e5fb7eeb3944688998fbc6f3625afbc P f3977d8fd8724b35b3d3cf52c96534bc [term 1 FOLLOWER]: Refusing update from remote peer ce7b4388b4584f4a912c49d57cdafc40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:54:18.633018 7423 consensus_queue.cc:1035] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3977d8fd8724b35b3d3cf52c96534bc" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:54:18.633567 7423 consensus_queue.cc:1035] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee5505bb889f40dd989938bda8954e15" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:54:18.704216 6894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:54:18.704548 6894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:54:18.708117 6894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 942053e138764c018d351c6936d86dd9: Sending DeleteTablet for 3 replicas of tablet 8e5fb7eeb3944688998fbc6f3625afbc
I20260806 10:54:18.709111 7032 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e5fb7eeb3944688998fbc6f3625afbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:54:18 UTC) from {username='oryx', principal='oryx/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:48114
I20260806 10:54:18.709308 7318 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e5fb7eeb3944688998fbc6f3625afbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:54:18 UTC) from {username='oryx', principal='oryx/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:46906
I20260806 10:54:18.709604 7486 tablet_replica.cc:330] T 8e5fb7eeb3944688998fbc6f3625afbc P ee5505bb889f40dd989938bda8954e15: stopping tablet replica
I20260806 10:54:18.709587 7175 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e5fb7eeb3944688998fbc6f3625afbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:54:18 UTC) from {username='oryx', principal='oryx/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:43534
I20260806 10:54:18.709663 7487 tablet_replica.cc:330] T 8e5fb7eeb3944688998fbc6f3625afbc P f3977d8fd8724b35b3d3cf52c96534bc: stopping tablet replica
I20260806 10:54:18.709825 7487 raft_consensus.cc:2229] T 8e5fb7eeb3944688998fbc6f3625afbc P f3977d8fd8724b35b3d3cf52c96534bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:54:18.709834 7486 raft_consensus.cc:2229] T 8e5fb7eeb3944688998fbc6f3625afbc P ee5505bb889f40dd989938bda8954e15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:54:18.709899 7488 tablet_replica.cc:330] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40: stopping tablet replica
I20260806 10:54:18.710042 7487 raft_consensus.cc:2258] T 8e5fb7eeb3944688998fbc6f3625afbc P f3977d8fd8724b35b3d3cf52c96534bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:54:18.710049 7486 raft_consensus.cc:2258] T 8e5fb7eeb3944688998fbc6f3625afbc P ee5505bb889f40dd989938bda8954e15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:54:18.710071 7488 raft_consensus.cc:2229] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:54:18.710417 7488 raft_consensus.cc:2258] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:54:18.711555 7488 ts_tablet_manager.cc:1905] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:54:18.711714 7486 ts_tablet_manager.cc:1905] T 8e5fb7eeb3944688998fbc6f3625afbc P ee5505bb889f40dd989938bda8954e15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:54:18.711948 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 6966
I20260806 10:54:18.718672 7488 ts_tablet_manager.cc:1918] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:54:18.718864 7488 log.cc:1196] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/8e5fb7eeb3944688998fbc6f3625afbc
I20260806 10:54:18.719390 7488 ts_tablet_manager.cc:1939] T 8e5fb7eeb3944688998fbc6f3625afbc P ce7b4388b4584f4a912c49d57cdafc40: Deleting consensus metadata
I20260806 10:54:18.719910 7486 ts_tablet_manager.cc:1918] T 8e5fb7eeb3944688998fbc6f3625afbc P ee5505bb889f40dd989938bda8954e15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:54:18.720192 7486 log.cc:1196] T 8e5fb7eeb3944688998fbc6f3625afbc P ee5505bb889f40dd989938bda8954e15: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-0/wal/wals/8e5fb7eeb3944688998fbc6f3625afbc
I20260806 10:54:18.720774 7486 ts_tablet_manager.cc:1939] T 8e5fb7eeb3944688998fbc6f3625afbc P ee5505bb889f40dd989938bda8954e15: Deleting consensus metadata
I20260806 10:54:18.720963 6881 catalog_manager.cc:4841] TS ce7b4388b4584f4a912c49d57cdafc40 (127.30.218.2:39917): tablet 8e5fb7eeb3944688998fbc6f3625afbc (table test-table [id=f0213beb51204686afd7ed53c4770fde]) successfully deleted
I20260806 10:54:18.722152 7487 ts_tablet_manager.cc:1905] T 8e5fb7eeb3944688998fbc6f3625afbc P f3977d8fd8724b35b3d3cf52c96534bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:54:18.722219 6880 catalog_manager.cc:4841] TS ee5505bb889f40dd989938bda8954e15 (127.30.218.1:38295): tablet 8e5fb7eeb3944688998fbc6f3625afbc (table test-table [id=f0213beb51204686afd7ed53c4770fde]) successfully deleted
I20260806 10:54:18.729514 7487 ts_tablet_manager.cc:1918] T 8e5fb7eeb3944688998fbc6f3625afbc P f3977d8fd8724b35b3d3cf52c96534bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 10:54:18.729696 7487 log.cc:1196] T 8e5fb7eeb3944688998fbc6f3625afbc P f3977d8fd8724b35b3d3cf52c96534bc: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/8e5fb7eeb3944688998fbc6f3625afbc
I20260806 10:54:18.730120 7487 ts_tablet_manager.cc:1939] T 8e5fb7eeb3944688998fbc6f3625afbc P f3977d8fd8724b35b3d3cf52c96534bc: Deleting consensus metadata
I20260806 10:54:18.731374 6878 catalog_manager.cc:4841] TS f3977d8fd8724b35b3d3cf52c96534bc (127.30.218.3:45881): tablet 8e5fb7eeb3944688998fbc6f3625afbc (table test-table [id=f0213beb51204686afd7ed53c4770fde]) successfully deleted
Aug 06 10:54:18 dist-test-slave-49h3 krb5kdc[6848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.218.1@KRBTEST.COM
I20260806 10:54:18.739399 7513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:18.714230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48124 (local address 127.30.218.1:38295)
0806 10:54:18.714416 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:18.714422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:18.724338 (+ 9916us) server_negotiation.cc:407] Connection header received
0806 10:54:18.724534 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:18.724539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:18.724628 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:18.724731 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:18.726079 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.726916 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.728174 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.728654 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.728776 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:18.733002 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:18.733022 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:18.733041 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:18.733089 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:18.736261 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:18.737034 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:18.737042 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:18.737044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:18.737213 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:18.737676 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:18.737681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:18.737684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:18.738095 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:18.738464 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:18.738710 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 10:54:18.739003 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 10:54:18.740141 7100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:18.868692 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 6966
I20260806 10:54:18.891049 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 7109
Aug 06 10:54:18 dist-test-slave-49h3 krb5kdc[6848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.218.2@KRBTEST.COM
I20260806 10:54:18.906592 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:18.891852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43568 (local address 127.30.218.2:39917)
0806 10:54:18.892019 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:18.892024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:18.892044 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:54:18.892204 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:18.892209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:18.892286 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:18.892371 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:18.893588 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.894352 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.895622 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:18.896093 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:18.896202 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:18.900558 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:18.900577 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:18.900597 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:18.900640 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:18.903799 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:18.904617 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:18.904623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:18.904626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:18.904765 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:18.905105 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:18.905108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:18.905110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:18.905533 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:18.905797 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:18.906071 (+ 274us) server_negotiation.cc:299] Negotiation successful
0806 10:54:18.906281 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 10:54:18.907343 7242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:19.041538 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 7109
I20260806 10:54:19.068375 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 7252
Aug 06 10:54:19 dist-test-slave-49h3 krb5kdc[6848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.218.3@KRBTEST.COM
I20260806 10:54:19.085304 7532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:19.069233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46924 (local address 127.30.218.3:45881)
0806 10:54:19.069742 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:19.069747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:19.069765 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:54:19.069829 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:19.069833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:19.069896 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:19.069995 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:19.071377 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:19.072173 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:19.073646 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:19.074064 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:19.074173 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:19.079156 (+ 4983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:19.079177 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:19.079195 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:19.079249 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:19.082309 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:19.083113 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:19.083120 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:19.083123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:19.083270 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:19.083688 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:19.083691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:19.083693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:19.084096 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:19.084365 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:19.084720 (+ 355us) server_negotiation.cc:299] Negotiation successful
0806 10:54:19.084931 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":283,"threads_started":1}
I20260806 10:54:19.086130 7384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:19.218639 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 7252
I20260806 10:54:19.218680 7534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:19.217889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:39917 (local address 127.30.218.3:39499)
0806 10:54:19.218295 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:19.218455 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.2:39917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":202,"threads_started":1}
I20260806 10:54:19.240273 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 6861
I20260806 10:54:19.241602 6934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:19.361711 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 6861
2026-08-06T10:54:19Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7104 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:54:19 dist-test-slave-49h3 krb5kdc[7539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:54:19 dist-test-slave-49h3 krb5kdc[7539](info): set up 2 sockets
Aug 06 10:54:19 dist-test-slave-49h3 krb5kdc[7539](info): commencing operation
krb5kdc: starting...
W20260806 10:54:20.435648 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.010s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:54:20 dist-test-slave-49h3 krb5kdc[7539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:54:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:54:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:54:20.747177 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:34005
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:36413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:34005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:54:21.108306 7552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:21.108702 7552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:21.108852 7552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:21.116506 7552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:54:21.116672 7552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:21.116762 7552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:21.116847 7552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:54:21.116915 7552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:54:21.127516 7552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:36413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:34005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:34005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:21.128971 7552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:21.130463 7552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:21.143280 7557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:21.143308 7558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:21.143909 7552 server_base.cc:1034] running on GCE node
W20260806 10:54:21.143994 7560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:21.145035 7552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:21.146682 7552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:21.147921 7552 hybrid_clock.cc:648] HybridClock initialized: now 1786013661147877 us; error 42 us; skew 500 ppm
I20260806 10:54:21.148288 7552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T10:54:21Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1851 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:54:21 dist-test-slave-49h3 krb5kdc[7566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:54:21 dist-test-slave-49h3 krb5kdc[7566](info): set up 2 sockets
Aug 06 10:54:21 dist-test-slave-49h3 krb5kdc[7566](info): commencing operation
krb5kdc: starting...
W20260806 10:54:22.291356 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.004s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:54:22 dist-test-slave-49h3 krb5kdc[7566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:54:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:54:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:54:22.597419 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:39573
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:39573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:54:22.949580 7579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:22.950107 7579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:22.950239 7579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:22.961217 7579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:54:22.961385 7579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:22.961488 7579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:22.961583 7579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:54:22.961648 7579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:54:22.973479 7579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:39573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:39573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:22.975306 7579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:22.976732 7579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:22.988816 7584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:22.988830 7585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:22.988768 7587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:22.989616 7579 server_base.cc:1034] running on GCE node
I20260806 10:54:22.990370 7579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:22.991683 7579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:22.992895 7579 hybrid_clock.cc:648] HybridClock initialized: now 1786013662992848 us; error 30 us; skew 500 ppm
I20260806 10:54:22.993253 7579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:22 dist-test-slave-49h3 krb5kdc[7566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013662, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:22.996353 7579 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:54:22.998030 7579 webserver.cc:460] Webserver started at http://127.30.218.62:34879/ using document root <none> and password file <none>
I20260806 10:54:22.998754 7579 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:22.998875 7579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:22.999274 7579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:23.003273 7579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "e7fddbce5fb043738aed1c5cd272bc76"
format_stamp: "Formatted at 2026-08-06 10:54:22 on dist-test-slave-49h3"
server_key: "75fa5d051bd36f3cac27fa88745613a4"
server_key_iv: "4a73c3acb487c2a33e67651ebcad733a"
server_key_version: "encryptionkey@0"
I20260806 10:54:23.004135 7579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "e7fddbce5fb043738aed1c5cd272bc76"
format_stamp: "Formatted at 2026-08-06 10:54:22 on dist-test-slave-49h3"
server_key: "75fa5d051bd36f3cac27fa88745613a4"
server_key_iv: "4a73c3acb487c2a33e67651ebcad733a"
server_key_version: "encryptionkey@0"
I20260806 10:54:23.010241 7579 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 10:54:23.014124 7594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:23.015681 7579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:54:23.015894 7579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "e7fddbce5fb043738aed1c5cd272bc76"
format_stamp: "Formatted at 2026-08-06 10:54:22 on dist-test-slave-49h3"
server_key: "75fa5d051bd36f3cac27fa88745613a4"
server_key_iv: "4a73c3acb487c2a33e67651ebcad733a"
server_key_version: "encryptionkey@0"
I20260806 10:54:23.016153 7579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:23.033821 7579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:23.038517 7579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:23.038846 7579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:23.057593 7579 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:39573
I20260806 10:54:23.057618 7646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:39573 every 8 connection(s)
I20260806 10:54:23.059038 7579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:54:23.063011 7647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:23.068676 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 7579
I20260806 10:54:23.068962 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:54:23.069828 31592 external_mini_cluster.cc:1428] Setting key 5fd0772f31f94516860dd0a25e7c398e
I20260806 10:54:23.075434 7647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76: Bootstrap starting.
I20260806 10:54:23.079174 7647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:23.080456 7647 log.cc:824] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76: Log is configured to *not* fsync() on all Append() calls
I20260806 10:54:23.084003 7647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76: No bootstrap required, opened a new log
Aug 06 10:54:23 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:54:23.092151 7647 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fddbce5fb043738aed1c5cd272bc76" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 39573 } }
I20260806 10:54:23.092552 7647 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:23.092651 7647 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7fddbce5fb043738aed1c5cd272bc76, State: Initialized, Role: FOLLOWER
I20260806 10:54:23.093310 7650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:23.072044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40558 (local address 127.30.218.62:39573)
0806 10:54:23.072901 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:23.072914 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:23.072958 (+ 44us) server_negotiation.cc:407] Connection header received
0806 10:54:23.074764 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:23.074789 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:23.075180 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:23.075509 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:23.077339 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:23.078662 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:23.080390 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:23.081292 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:23.081519 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:23.086141 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:23.086149 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:23.086163 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:23.086203 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:23.089258 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:23.089967 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:23.089976 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:23.089982 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:23.090147 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:23.090518 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:23.090522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:23.090524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:23.091062 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:23.091392 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:23.091797 (+ 405us) server_negotiation.cc:299] Negotiation successful
0806 10:54:23.092198 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":385,"threads_started":1}
I20260806 10:54:23.093204 7647 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fddbce5fb043738aed1c5cd272bc76" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 39573 } }
I20260806 10:54:23.093482 7647 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:54:23.093628 7647 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:54:23.093796 7647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:23.096120 7647 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fddbce5fb043738aed1c5cd272bc76" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 39573 } }
I20260806 10:54:23.096607 7647 leader_election.cc:302] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7fddbce5fb043738aed1c5cd272bc76; no voters:
I20260806 10:54:23.097152 7647 leader_election.cc:288] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:54:23.097272 7652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:54:23.097961 7652 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [term 1 LEADER]: Becoming Leader. State: Replica: e7fddbce5fb043738aed1c5cd272bc76, State: Running, Role: LEADER
I20260806 10:54:23.098475 7652 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fddbce5fb043738aed1c5cd272bc76" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 39573 } }
I20260806 10:54:23.099165 7647 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:54:23.100863 7653 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7fddbce5fb043738aed1c5cd272bc76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fddbce5fb043738aed1c5cd272bc76" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 39573 } } }
I20260806 10:54:23.101264 7653 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [sys.catalog]: This master's current role is: LEADER
I20260806 10:54:23.102083 7654 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7fddbce5fb043738aed1c5cd272bc76. Latest consensus state: current_term: 1 leader_uuid: "e7fddbce5fb043738aed1c5cd272bc76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fddbce5fb043738aed1c5cd272bc76" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 39573 } } }
I20260806 10:54:23.102370 7654 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [sys.catalog]: This master's current role is: LEADER
I20260806 10:54:23.106694 7657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:54:23.113723 7657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:54:23.124498 7657 catalog_manager.cc:1347] Generated new cluster ID: 636322b2564744b5ba17a816481e4bb7
I20260806 10:54:23.124624 7657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:54:23.138931 7657 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:54:23.140947 7657 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:54:23.151742 7657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76: Generated new TSK 0
I20260806 10:54:23.152994 7657 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
I20260806 10:54:23.221626 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:39573
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:54:23.575590 7673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:23.575922 7673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:23.576056 7673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:23.583470 7673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:23.583619 7673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:23.583770 7673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:54:23.594050 7673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:39573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:23.595495 7673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:23.596928 7673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:23.611069 7681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:23.611141 7679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:23.611039 7678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:23.612092 7673 server_base.cc:1034] running on GCE node
I20260806 10:54:23.612825 7673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:23.613978 7673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:23.615221 7673 hybrid_clock.cc:648] HybridClock initialized: now 1786013663615143 us; error 56 us; skew 500 ppm
I20260806 10:54:23.615586 7673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:23 dist-test-slave-49h3 krb5kdc[7566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013663, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:23.619155 7673 init.cc:377] Logged in from keytab as kudu/127.30.218.1@KRBTEST.COM (short username kudu)
I20260806 10:54:23.620966 7673 webserver.cc:460] Webserver started at http://127.30.218.1:36921/ using document root <none> and password file <none>
I20260806 10:54:23.621817 7673 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:23.621946 7673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:23.622292 7673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:23.625520 7673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "cc15cdeec7d744e3a065fd395f2fc5d9"
format_stamp: "Formatted at 2026-08-06 10:54:23 on dist-test-slave-49h3"
server_key: "1175a152f1f1dff85b518dc9f89a3fe9"
server_key_iv: "813494a137ae3e8e57c1cdb7b75ad8e2"
server_key_version: "encryptionkey@0"
I20260806 10:54:23.626441 7673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "cc15cdeec7d744e3a065fd395f2fc5d9"
format_stamp: "Formatted at 2026-08-06 10:54:23 on dist-test-slave-49h3"
server_key: "1175a152f1f1dff85b518dc9f89a3fe9"
server_key_iv: "813494a137ae3e8e57c1cdb7b75ad8e2"
server_key_version: "encryptionkey@0"
I20260806 10:54:23.632311 7673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:54:23.636365 7688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:23.637993 7673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:54:23.638159 7673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "cc15cdeec7d744e3a065fd395f2fc5d9"
format_stamp: "Formatted at 2026-08-06 10:54:23 on dist-test-slave-49h3"
server_key: "1175a152f1f1dff85b518dc9f89a3fe9"
server_key_iv: "813494a137ae3e8e57c1cdb7b75ad8e2"
server_key_version: "encryptionkey@0"
I20260806 10:54:23.638399 7673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:23.651176 7673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:23.655651 7673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:23.655946 7673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:23.656924 7673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:54:23.658361 7673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:23.658442 7673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:23.658560 7673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:23.658614 7673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:23.690716 7673 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:44571
I20260806 10:54:23.690762 7801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:44571 every 8 connection(s)
I20260806 10:54:23.692058 7673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:54:23.693642 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 7673
I20260806 10:54:23.693878 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:54:23.694693 31592 external_mini_cluster.cc:1428] Setting key 3b5f8b78dbdbf5d2717ba7e3d2b015c3
Aug 06 10:54:23 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013663, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:54:23.719822 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:23.695700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:46409 (local address 127.30.218.62:39573)
0806 10:54:23.697594 (+ 1894us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:23.697601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:23.698799 (+ 1198us) server_negotiation.cc:407] Connection header received
0806 10:54:23.701560 (+ 2761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:23.701566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:23.701673 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:23.701822 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:23.704127 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:23.704896 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:23.707294 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:23.707856 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:23.707978 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:23.712581 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:23.712589 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:23.712595 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:23.712641 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:23.716140 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:23.717108 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:23.717114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:23.717117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:23.717298 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:23.717843 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:23.717847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:23.717850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:23.718231 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:23.718396 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:23.719164 (+ 768us) server_negotiation.cc:299] Negotiation successful
0806 10:54:23.719440 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1713,"thread_start_us":1916,"threads_started":1}
I20260806 10:54:23.720754 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:23.696432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.1:46409)
0806 10:54:23.698613 (+ 2181us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:23.698672 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:23.701138 (+ 2466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:23.702164 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:23.702178 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:23.702991 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:23.703890 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:23.703921 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:23.705117 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:23.705123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:23.707111 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:23.707130 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:23.708141 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:23.708149 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:23.708465 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:23.709257 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:23.709288 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:23.712249 (+ 2961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:23.716378 (+ 4129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:23.716388 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:23.716404 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:23.716933 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:23.717406 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:23.717409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:23.717411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:23.717691 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:23.718346 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:23.718353 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:23.718726 (+ 373us) client_negotiation.cc:769] Sending connection context
0806 10:54:23.718970 (+ 244us) client_negotiation.cc:240] Negotiation successful
0806 10:54:23.719448 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1824,"thread_start_us":1544,"threads_started":1}
I20260806 10:54:23.722721 7802 heartbeater.cc:347] Connected to a master server at 127.30.218.62:39573
I20260806 10:54:23.723088 7802 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:23.723906 7802 heartbeater.cc:511] Master 127.30.218.62:39573 requested a full tablet report, sending...
I20260806 10:54:23.726476 7611 ts_manager.cc:194] Registered new tserver with Master: cc15cdeec7d744e3a065fd395f2fc5d9 (127.30.218.1:44571)
WARNING: no policy specified for kudu/127.30.218.2@KRBTEST.COM; defaulting to no policy
I20260806 10:54:23.728122 7611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.1@KRBTEST.COM'} at 127.30.218.1:46409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
I20260806 10:54:23.764942 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:39573
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:54:24.135569 7808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:24.135903 7808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:24.136049 7808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:24.143783 7808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:24.143936 7808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:24.144125 7808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:54:24.154759 7808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:39573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:24.156173 7808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:24.157560 7808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:24.171598 7814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:24.171522 7816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:24.171581 7813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:24.172303 7808 server_base.cc:1034] running on GCE node
I20260806 10:54:24.172979 7808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:24.174120 7808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:24.175364 7808 hybrid_clock.cc:648] HybridClock initialized: now 1786013664175306 us; error 48 us; skew 500 ppm
I20260806 10:54:24.175707 7808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:24 dist-test-slave-49h3 krb5kdc[7566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013664, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:24.179136 7808 init.cc:377] Logged in from keytab as kudu/127.30.218.2@KRBTEST.COM (short username kudu)
I20260806 10:54:24.180882 7808 webserver.cc:460] Webserver started at http://127.30.218.2:44469/ using document root <none> and password file <none>
I20260806 10:54:24.181732 7808 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:24.181872 7808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:24.182212 7808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:24.185779 7808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "33af5b52ad82419aaeb5ea721b24e48b"
format_stamp: "Formatted at 2026-08-06 10:54:24 on dist-test-slave-49h3"
server_key: "c476f2e33e211d938754789e9992d0ab"
server_key_iv: "a36a475a607f9784b81cd62499ae6281"
server_key_version: "encryptionkey@0"
I20260806 10:54:24.186717 7808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "33af5b52ad82419aaeb5ea721b24e48b"
format_stamp: "Formatted at 2026-08-06 10:54:24 on dist-test-slave-49h3"
server_key: "c476f2e33e211d938754789e9992d0ab"
server_key_iv: "a36a475a607f9784b81cd62499ae6281"
server_key_version: "encryptionkey@0"
I20260806 10:54:24.192623 7808 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:54:24.196650 7823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:24.198319 7808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 10:54:24.198513 7808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "33af5b52ad82419aaeb5ea721b24e48b"
format_stamp: "Formatted at 2026-08-06 10:54:24 on dist-test-slave-49h3"
server_key: "c476f2e33e211d938754789e9992d0ab"
server_key_iv: "a36a475a607f9784b81cd62499ae6281"
server_key_version: "encryptionkey@0"
I20260806 10:54:24.198737 7808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:24.216697 7808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:24.221287 7808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:24.221642 7808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:24.222678 7808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:54:24.224182 7808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:24.224273 7808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:24.224407 7808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:24.224462 7808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:24.256058 7808 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:35741
I20260806 10:54:24.256098 7939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:35741 every 8 connection(s)
I20260806 10:54:24.257364 7808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:54:24.266708 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 7808
I20260806 10:54:24.266883 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:54:24.267315 31592 external_mini_cluster.cc:1428] Setting key ee5cd8c9140b37b9ad7e52b4b3b8fa81
Aug 06 10:54:24 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013664, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:54:24.277379 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:24.259883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:39573 (local address 127.30.218.62:39573)
0806 10:54:24.260377 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:24.260384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:24.261375 (+ 991us) server_negotiation.cc:407] Connection header received
0806 10:54:24.263176 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:24.263181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:24.263259 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:24.263357 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:24.264859 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:24.265661 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:24.267304 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:24.267798 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:24.267918 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:24.271582 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:24.271588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:24.271591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:24.271626 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:24.274246 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:24.275010 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:24.275014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:24.275017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:24.275150 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:24.275591 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:24.275595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:24.275597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:24.275870 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:24.276008 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:24.276758 (+ 750us) server_negotiation.cc:299] Negotiation successful
0806 10:54:24.277008 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":214,"threads_started":1}
I20260806 10:54:24.278470 7943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:24.260413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.2:39573)
0806 10:54:24.261207 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:24.261244 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:24.262926 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:24.263555 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:24.263562 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:24.264102 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:24.264683 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:24.264701 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:24.265821 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:24.265825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:24.267153 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:24.267166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:24.268043 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:24.268051 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:24.268278 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:24.269000 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:24.269024 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:24.271324 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:24.274406 (+ 3082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:24.274416 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:24.274435 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:24.274885 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:24.275265 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:24.275270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:24.275274 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:24.275469 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:24.276002 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:24.276011 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:24.276385 (+ 374us) client_negotiation.cc:769] Sending connection context
0806 10:54:24.276656 (+ 271us) client_negotiation.cc:240] Negotiation successful
0806 10:54:24.277237 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":305,"threads_started":1}
I20260806 10:54:24.280184 7940 heartbeater.cc:347] Connected to a master server at 127.30.218.62:39573
I20260806 10:54:24.280514 7940 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:24.281353 7940 heartbeater.cc:511] Master 127.30.218.62:39573 requested a full tablet report, sending...
I20260806 10:54:24.283167 7611 ts_manager.cc:194] Registered new tserver with Master: 33af5b52ad82419aaeb5ea721b24e48b (127.30.218.2:35741)
I20260806 10:54:24.284312 7611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.2@KRBTEST.COM'} at 127.30.218.2:39573
WARNING: no policy specified for kudu/127.30.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
I20260806 10:54:24.333072 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:39573
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:54:24.690533 7946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:24.690856 7946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:24.690989 7946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:24.698611 7946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:24.698740 7946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:24.698899 7946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:54:24.709316 7946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:39573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:24.710685 7946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:24.712005 7946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:24.726184 7952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:24.726394 7951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:24.726202 7954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:24.727624 7946 server_base.cc:1034] running on GCE node
I20260806 10:54:24.728318 7946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:24.729372 7946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:24.730665 7946 hybrid_clock.cc:648] HybridClock initialized: now 1786013664730600 us; error 48 us; skew 500 ppm
I20260806 10:54:24.731026 7946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:54:24.731616 7802 heartbeater.cc:503] Master 127.30.218.62:39573 was elected leader, sending a full tablet report...
Aug 06 10:54:24 dist-test-slave-49h3 krb5kdc[7566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013664, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:24.734890 7946 init.cc:377] Logged in from keytab as kudu/127.30.218.3@KRBTEST.COM (short username kudu)
I20260806 10:54:24.736748 7946 webserver.cc:460] Webserver started at http://127.30.218.3:34073/ using document root <none> and password file <none>
I20260806 10:54:24.737609 7946 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:24.737748 7946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:24.738065 7946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:24.741274 7946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "d4c473d8e0df44d3bc3db91eac42c712"
format_stamp: "Formatted at 2026-08-06 10:54:24 on dist-test-slave-49h3"
server_key: "52aa9ef39c3e8cdd5a0a5d501aa25cfc"
server_key_iv: "5301f9baec3d1a3eda895bfc0d3b6345"
server_key_version: "encryptionkey@0"
I20260806 10:54:24.742290 7946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "d4c473d8e0df44d3bc3db91eac42c712"
format_stamp: "Formatted at 2026-08-06 10:54:24 on dist-test-slave-49h3"
server_key: "52aa9ef39c3e8cdd5a0a5d501aa25cfc"
server_key_iv: "5301f9baec3d1a3eda895bfc0d3b6345"
server_key_version: "encryptionkey@0"
I20260806 10:54:24.748232 7946 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260806 10:54:24.752380 7961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:24.754068 7946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:54:24.754268 7946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "d4c473d8e0df44d3bc3db91eac42c712"
format_stamp: "Formatted at 2026-08-06 10:54:24 on dist-test-slave-49h3"
server_key: "52aa9ef39c3e8cdd5a0a5d501aa25cfc"
server_key_iv: "5301f9baec3d1a3eda895bfc0d3b6345"
server_key_version: "encryptionkey@0"
I20260806 10:54:24.754503 7946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:24.769906 7946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:24.774493 7946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:24.774807 7946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:24.775859 7946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:54:24.777320 7946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:24.777412 7946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:24.777580 7946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:24.777639 7946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:24.810746 7946 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:39221
I20260806 10:54:24.810783 8076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:39221 every 8 connection(s)
I20260806 10:54:24.812111 7946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:54:24.814797 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 7946
I20260806 10:54:24.815008 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:54:24.815593 31592 external_mini_cluster.cc:1428] Setting key 7880b4d9b614a6f77020777a308876d6
Aug 06 10:54:24 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013664, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:54:24.833658 8079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:24.814874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:52599 (local address 127.30.218.62:39573)
0806 10:54:24.815391 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:24.815397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:24.816500 (+ 1103us) server_negotiation.cc:407] Connection header received
0806 10:54:24.818635 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:24.818641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:24.818744 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:24.818838 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:24.820555 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:24.821384 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:24.823053 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:24.823486 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:24.823611 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:24.827471 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:24.827477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:24.827480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:24.827518 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:24.830301 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:24.831327 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:24.831333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:24.831337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:24.831499 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:24.831959 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:24.831962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:24.831964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:24.832272 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:24.832397 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:24.833106 (+ 709us) server_negotiation.cc:299] Negotiation successful
0806 10:54:24.833311 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":230,"threads_started":1}
I20260806 10:54:24.834641 8080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:24.815442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.3:52599)
0806 10:54:24.816323 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:24.816379 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:24.818365 (+ 1986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:24.819103 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:24.819113 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:24.819717 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:24.820378 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:24.820405 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:24.821566 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:24.821571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:24.822875 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:24.822889 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:24.823750 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:24.823756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:24.823998 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:24.824726 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:24.824753 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:24.827198 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:24.830479 (+ 3281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:24.830490 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:24.830512 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:24.831144 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:24.831631 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:24.831635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:24.831637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:24.831863 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:24.832393 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:24.832400 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:24.832781 (+ 381us) client_negotiation.cc:769] Sending connection context
0806 10:54:24.833031 (+ 250us) client_negotiation.cc:240] Negotiation successful
0806 10:54:24.833524 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":361,"threads_started":1}
I20260806 10:54:24.836151 8077 heartbeater.cc:347] Connected to a master server at 127.30.218.62:39573
I20260806 10:54:24.836488 8077 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:24.837275 8077 heartbeater.cc:511] Master 127.30.218.62:39573 requested a full tablet report, sending...
I20260806 10:54:24.838819 7611 ts_manager.cc:194] Registered new tserver with Master: d4c473d8e0df44d3bc3db91eac42c712 (127.30.218.3:39221)
I20260806 10:54:24.839632 7611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.3@KRBTEST.COM'} at 127.30.218.3:52599
I20260806 10:54:24.843856 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.30.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.1.keytab.
Entry for principal oryx/127.30.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.1.keytab.
WARNING: no policy specified for oryx/127.30.218.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.2.keytab.
Entry for principal oryx/127.30.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.2.keytab.
WARNING: no policy specified for oryx/127.30.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.3.keytab.
Entry for principal oryx/127.30.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.3.keytab.
I20260806 10:54:25.028587 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 7673
Aug 06 10:54:25 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.1@KRBTEST.COM
I20260806 10:54:25.046590 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:25.029745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58436 (local address 127.30.218.1:44571)
0806 10:54:25.030409 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:25.030417 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:25.030447 (+ 30us) server_negotiation.cc:407] Connection header received
0806 10:54:25.030587 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:25.030594 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:25.030841 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:25.030986 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:25.032448 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:25.033622 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:25.034981 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:25.035601 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:25.035708 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:25.040276 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:25.040294 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:25.040312 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:25.040356 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:25.043503 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:25.044382 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:25.044390 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:25.044392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:25.044544 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:25.044997 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:25.045001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:25.045004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:25.045436 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:25.045742 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:25.046039 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 10:54:25.046281 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":311,"threads_started":1}
I20260806 10:54:25.047919 7797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:25.134661 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 7673
I20260806 10:54:25.152870 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 7808
Aug 06 10:54:25 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.2@KRBTEST.COM
I20260806 10:54:25.170873 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:25.153872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40376 (local address 127.30.218.2:35741)
0806 10:54:25.154477 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:25.154485 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:25.154512 (+ 27us) server_negotiation.cc:407] Connection header received
0806 10:54:25.154584 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:25.154591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:25.154819 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:25.154954 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:25.156359 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:25.157609 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:25.159027 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:25.159642 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:25.159776 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:25.164448 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:25.164468 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:25.164486 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:25.164536 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:25.167600 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:25.168451 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:25.168460 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:25.168462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:25.168612 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:25.169015 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:25.169019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:25.169022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:25.169425 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:25.169796 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:25.170163 (+ 367us) server_negotiation.cc:299] Negotiation successful
0806 10:54:25.170503 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":326,"threads_started":1}
I20260806 10:54:25.172304 7935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:25.256983 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 7808
I20260806 10:54:25.273214 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 7946
Aug 06 10:54:25 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.3@KRBTEST.COM
I20260806 10:54:25.290983 8093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:25.274268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47114 (local address 127.30.218.3:39221)
0806 10:54:25.274780 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:25.274787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:25.274816 (+ 29us) server_negotiation.cc:407] Connection header received
0806 10:54:25.274880 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:25.274887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:25.275100 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:25.275236 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:25.276593 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:25.277792 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:25.279102 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:25.279698 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:25.279804 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:25.284495 (+ 4691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:25.284512 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:25.284540 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:25.284584 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:25.288056 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:25.288913 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:25.288921 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:25.288924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:25.289070 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:25.289394 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:25.289397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:25.289399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:25.289890 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:25.290137 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:25.290408 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 10:54:25.290649 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":226,"threads_started":1}
I20260806 10:54:25.292451 8072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:25.377182 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 7946
I20260806 10:54:25.393658 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 7579
I20260806 10:54:25.394956 7642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:25.502127 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 7579
I20260806 10:54:25.523717 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:39573
--webserver_interface=127.30.218.62
--webserver_port=34879
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:39573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:54:25.881436 8096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:25.881882 8096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:25.882038 8096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:25.889885 8096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:54:25.890058 8096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:25.890148 8096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:25.890233 8096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:54:25.890298 8096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:54:25.901074 8096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:39573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:39573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=34879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:25.902621 8096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:25.904098 8096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:25.917012 8101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:25.916949 8102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:25.916934 8104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:25.917958 8096 server_base.cc:1034] running on GCE node
I20260806 10:54:25.918820 8096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:25.920467 8096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:25.921693 8096 hybrid_clock.cc:648] HybridClock initialized: now 1786013665921618 us; error 60 us; skew 500 ppm
I20260806 10:54:25.922071 8096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:25 dist-test-slave-49h3 krb5kdc[7566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013665, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:25.926911 8096 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:54:25.928918 8096 webserver.cc:460] Webserver started at http://127.30.218.62:34879/ using document root <none> and password file <none>
I20260806 10:54:25.929805 8096 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:25.929949 8096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:25.937144 8096 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 10:54:25.941030 8111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:25.942910 8096 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 10:54:25.943107 8096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "e7fddbce5fb043738aed1c5cd272bc76"
format_stamp: "Formatted at 2026-08-06 10:54:22 on dist-test-slave-49h3"
server_key: "75fa5d051bd36f3cac27fa88745613a4"
server_key_iv: "4a73c3acb487c2a33e67651ebcad733a"
server_key_version: "encryptionkey@0"
I20260806 10:54:25.944310 8096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:25.967973 8096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:25.972445 8096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:25.972692 8096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:25.991940 8096 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:39573
I20260806 10:54:25.991961 8162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:39573 every 8 connection(s)
I20260806 10:54:25.993660 8096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:54:25.995546 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 8096
I20260806 10:54:25.996627 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:44571
--local_ip_for_outbound_sockets=127.30.218.1
--tserver_master_addrs=127.30.218.62:39573
--webserver_port=36921
--webserver_interface=127.30.218.1
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.30.218.1
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:54:26.003579 8163 sys_catalog.cc:263] Verifying existing consensus state
I20260806 10:54:26.008445 8163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76: Bootstrap starting.
I20260806 10:54:26.029837 8163 log.cc:824] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76: Log is configured to *not* fsync() on all Append() calls
I20260806 10:54:26.041157 8163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 10:54:26.041785 8163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76: Bootstrap complete.
I20260806 10:54:26.051379 8163 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fddbce5fb043738aed1c5cd272bc76" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 39573 } }
I20260806 10:54:26.052255 8163 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7fddbce5fb043738aed1c5cd272bc76, State: Initialized, Role: FOLLOWER
I20260806 10:54:26.052898 8163 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fddbce5fb043738aed1c5cd272bc76" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 39573 } }
I20260806 10:54:26.053134 8163 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:54:26.053263 8163 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:54:26.053427 8163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [term 1 FOLLOWER]: Advancing to term 2
I20260806 10:54:26.057477 8163 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fddbce5fb043738aed1c5cd272bc76" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 39573 } }
I20260806 10:54:26.058005 8163 leader_election.cc:302] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7fddbce5fb043738aed1c5cd272bc76; no voters:
I20260806 10:54:26.058629 8163 leader_election.cc:288] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 10:54:26.058782 8167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 10:54:26.059880 8167 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [term 2 LEADER]: Becoming Leader. State: Replica: e7fddbce5fb043738aed1c5cd272bc76, State: Running, Role: LEADER
I20260806 10:54:26.060508 8167 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fddbce5fb043738aed1c5cd272bc76" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 39573 } }
I20260806 10:54:26.060848 8163 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:54:26.063035 8169 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7fddbce5fb043738aed1c5cd272bc76. Latest consensus state: current_term: 2 leader_uuid: "e7fddbce5fb043738aed1c5cd272bc76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fddbce5fb043738aed1c5cd272bc76" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 39573 } } }
I20260806 10:54:26.063393 8169 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [sys.catalog]: This master's current role is: LEADER
I20260806 10:54:26.063122 8168 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e7fddbce5fb043738aed1c5cd272bc76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fddbce5fb043738aed1c5cd272bc76" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 39573 } } }
I20260806 10:54:26.063503 8168 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76 [sys.catalog]: This master's current role is: LEADER
I20260806 10:54:26.073344 8174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:54:26.079619 8174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:54:26.081246 8174 catalog_manager.cc:1259] Loaded cluster ID: 636322b2564744b5ba17a816481e4bb7
I20260806 10:54:26.081398 8174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:54:26.085639 8174 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:54:26.087260 8174 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e7fddbce5fb043738aed1c5cd272bc76: Loaded TSK: 0
I20260806 10:54:26.088876 8174 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 10:54:26.386003 8165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:26.386356 8165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:26.386489 8165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:26.394284 8165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:26.394413 8165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:26.394572 8165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:54:26.405148 8165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.1.keytab
--principal=oryx/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:44571
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=36921
--tserver_master_addrs=127.30.218.62:39573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:26.406653 8165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:26.408039 8165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:26.423772 8191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:26.423779 8192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:26.424835 8194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:26.425032 8165 server_base.cc:1034] running on GCE node
I20260806 10:54:26.425791 8165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:26.426975 8165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:26.428226 8165 hybrid_clock.cc:648] HybridClock initialized: now 1786013666428161 us; error 66 us; skew 500 ppm
I20260806 10:54:26.428566 8165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:26 dist-test-slave-49h3 krb5kdc[7566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013666, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:26.432168 8165 init.cc:377] Logged in from keytab as oryx/127.30.218.1@KRBTEST.COM (short username oryx)
I20260806 10:54:26.434048 8165 webserver.cc:460] Webserver started at http://127.30.218.1:36921/ using document root <none> and password file <none>
I20260806 10:54:26.434849 8165 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:26.434994 8165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:26.441298 8165 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 10:54:26.444784 8201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:26.446491 8165 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 10:54:26.446686 8165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "cc15cdeec7d744e3a065fd395f2fc5d9"
format_stamp: "Formatted at 2026-08-06 10:54:23 on dist-test-slave-49h3"
server_key: "1175a152f1f1dff85b518dc9f89a3fe9"
server_key_iv: "813494a137ae3e8e57c1cdb7b75ad8e2"
server_key_version: "encryptionkey@0"
I20260806 10:54:26.447825 8165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:26.467638 8165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:26.472385 8165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:26.472692 8165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:26.473858 8165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:54:26.475389 8165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:26.475492 8165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:26.475627 8165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:26.475700 8165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:26.508400 8165 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:44571
I20260806 10:54:26.508456 8313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:44571 every 8 connection(s)
I20260806 10:54:26.509910 8165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:54:26.517204 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 8165
I20260806 10:54:26.518014 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:35741
--local_ip_for_outbound_sockets=127.30.218.2
--tserver_master_addrs=127.30.218.62:39573
--webserver_port=44469
--webserver_interface=127.30.218.2
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.30.218.2
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 10:54:26 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:26 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 10:54:26.889081 8318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:26.889477 8318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:26.889626 8318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:26.897852 8318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:26.898018 8318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:26.898190 8318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:54:26.908668 8318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.2.keytab
--principal=oryx/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:35741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=44469
--tserver_master_addrs=127.30.218.62:39573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:26.910197 8318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:26.911607 8318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:26.928756 8329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:26.928923 8326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:26.929719 8325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:26.929764 8318 server_base.cc:1034] running on GCE node
I20260806 10:54:26.930801 8318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:26.932397 8318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:26.933688 8318 hybrid_clock.cc:648] HybridClock initialized: now 1786013666933597 us; error 75 us; skew 500 ppm
I20260806 10:54:26.934196 8318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:26 dist-test-slave-49h3 krb5kdc[7566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013666, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:26.939460 8318 init.cc:377] Logged in from keytab as oryx/127.30.218.2@KRBTEST.COM (short username oryx)
I20260806 10:54:26.941952 8318 webserver.cc:460] Webserver started at http://127.30.218.2:44469/ using document root <none> and password file <none>
I20260806 10:54:26.943164 8318 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:26.943372 8318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:26.952471 8318 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 10:54:26.956626 8336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:26.958758 8318 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260806 10:54:26.959111 8318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "33af5b52ad82419aaeb5ea721b24e48b"
format_stamp: "Formatted at 2026-08-06 10:54:24 on dist-test-slave-49h3"
server_key: "c476f2e33e211d938754789e9992d0ab"
server_key_iv: "a36a475a607f9784b81cd62499ae6281"
server_key_version: "encryptionkey@0"
I20260806 10:54:26.960947 8318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:27.004832 8318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:27.011857 8318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:27.012264 8318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:27.014559 8318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:54:27.016778 8318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:27.016923 8318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:27.017071 8318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:27.017166 8318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:27.054579 8318 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:35741
I20260806 10:54:27.054626 8448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:35741 every 8 connection(s)
I20260806 10:54:27.057063 8318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:54:27.065032 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 8318
I20260806 10:54:27.065912 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:39221
--local_ip_for_outbound_sockets=127.30.218.3
--tserver_master_addrs=127.30.218.62:39573
--webserver_port=34073
--webserver_interface=127.30.218.3
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.30.218.3
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 10:54:27 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:27 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 10:54:27.573941 8317 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.049s user 0.010s sys 0.007s
I20260806 10:54:27.576032 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:26.513032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.1:44899)
0806 10:54:26.513911 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:26.513950 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:26.515723 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:26.517523 (+ 1800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:26.517532 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:26.518135 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:26.518762 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:26.518783 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:26.521188 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:26.521193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:26.522646 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:26.522664 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:26.524148 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:26.524154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:26.524393 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:26.525175 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:26.525202 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:27.574819 (+1049617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":390,"threads_started":1}
W20260806 10:54:27.576015 8316 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:26.512941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:44899 (local address 127.30.218.62:39573)
0806 10:54:26.513812 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:26.513828 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:26.514087 (+ 259us) server_negotiation.cc:407] Connection header received
0806 10:54:26.516347 (+ 2260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:26.516361 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:26.516825 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:26.517288 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:26.519024 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:26.520905 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:26.522836 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:26.523866 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:26.524080 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:27.575249 (+1051169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.1:44899: BlockingRecv error: recv got EOF from 127.30.218.1:44899 (error 108)
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":339,"threads_started":1}
W20260806 10:54:27.577283 8314 heartbeater.cc:644] Failed to heartbeat to 127.30.218.62:39573 (0 consecutive failures): Not authorized: Failed to ping master at 127.30.218.62:39573: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Aug 06 10:54:27 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:27 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:27.587177 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:27.578065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.1:53077)
0806 10:54:27.578353 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:27.578374 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:27.578507 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:27.578859 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:27.578864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:27.579263 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:27.579433 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:27.579442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:27.580501 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:27.580505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:27.581553 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:27.581568 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:27.582310 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:27.582317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:27.582472 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:27.583121 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:27.583140 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:27.586862 (+ 3722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164}
W20260806 10:54:27.587375 8316 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:27.578216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:53077 (local address 127.30.218.62:39573)
0806 10:54:27.578375 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:27.578381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:27.578432 (+ 51us) server_negotiation.cc:407] Connection header received
0806 10:54:27.578646 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:27.578651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:27.578716 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:27.578800 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:27.579601 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:27.580339 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:27.581701 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:27.582117 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:27.582235 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:27.587145 (+ 4910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.1:53077: BlockingRecv error: recv got EOF from 127.30.218.1:53077 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 10:54:27 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:27 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:27.597095 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:27.588235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.1:47099)
0806 10:54:27.588395 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:27.588416 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:27.588762 (+ 346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:27.589112 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:27.589117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:27.589494 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:27.589672 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:27.589683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:27.590740 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:27.590744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:27.591708 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:27.591722 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:27.592423 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:27.592430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:27.592596 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:27.593184 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:27.593203 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:27.596789 (+ 3586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 10:54:27.597288 8316 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:27.588609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:47099 (local address 127.30.218.62:39573)
0806 10:54:27.588772 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:27.588777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:27.588797 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:54:27.588887 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:27.588893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:27.588976 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:27.589059 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:27.589823 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:27.590589 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:27.591857 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:27.592269 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:27.592378 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:27.597066 (+ 4688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.1:47099: BlockingRecv error: recv got EOF from 127.30.218.1:47099 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260806 10:54:27.597831 8314 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 10:54:27.652820 8453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:27.653362 8453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:27.653585 8453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:27.667474 8453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:27.667706 8453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:27.667936 8453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:54:27.686738 8453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/krb5kdc/oryx_127.30.218.3.keytab
--principal=oryx/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:39221
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=34073
--tserver_master_addrs=127.30.218.62:39573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:27.688982 8453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:27.691211 8453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:27.709815 8459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:27.709815 8460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:27.710883 8463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:27.712366 8453 server_base.cc:1034] running on GCE node
I20260806 10:54:27.713091 8453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:27.714262 8453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:27.715613 8453 hybrid_clock.cc:648] HybridClock initialized: now 1786013667715547 us; error 81 us; skew 500 ppm
I20260806 10:54:27.715981 8453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:27 dist-test-slave-49h3 krb5kdc[7566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013667, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:27.719846 8453 init.cc:377] Logged in from keytab as oryx/127.30.218.3@KRBTEST.COM (short username oryx)
I20260806 10:54:27.722426 8453 webserver.cc:460] Webserver started at http://127.30.218.3:34073/ using document root <none> and password file <none>
I20260806 10:54:27.723722 8453 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:27.723932 8453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:27.733335 8453 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 10:54:27.738010 8470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:27.740331 8453 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260806 10:54:27.740656 8453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "d4c473d8e0df44d3bc3db91eac42c712"
format_stamp: "Formatted at 2026-08-06 10:54:24 on dist-test-slave-49h3"
server_key: "52aa9ef39c3e8cdd5a0a5d501aa25cfc"
server_key_iv: "5301f9baec3d1a3eda895bfc0d3b6345"
server_key_version: "encryptionkey@0"
I20260806 10:54:27.742575 8453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:27.768502 8453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:27.775629 8453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:27.776038 8453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:27.777602 8453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:54:27.779906 8453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:27.780052 8453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:27.780194 8453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:27.780279 8453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:27.820158 8453 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:39221
I20260806 10:54:27.820191 8583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:39221 every 8 connection(s)
I20260806 10:54:27.821826 8453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:54:27.823385 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 8453
Aug 06 10:54:27 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:27 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:27.848055 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:27.825357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50684 (local address 127.30.218.62:39573)
0806 10:54:27.825868 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:27.825876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:27.829096 (+ 3220us) server_negotiation.cc:407] Connection header received
0806 10:54:27.829353 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:27.829361 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:27.829514 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:27.829665 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:27.832089 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:27.832849 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:27.834308 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:27.834985 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:27.835103 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:27.838429 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:27.838444 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:27.838472 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:27.838566 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:27.843685 (+ 5119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:27.844705 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:27.844713 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:27.844722 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:27.844948 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:27.845539 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:27.845545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:27.845548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:27.846209 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:27.846543 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:27.847210 (+ 667us) server_negotiation.cc:299] Negotiation successful
0806 10:54:27.847664 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340}
W20260806 10:54:28.210625 8452 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.135s user 0.008s sys 0.014s
W20260806 10:54:28.212297 8451 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:27.060602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:47643 (local address 127.30.218.62:39573)
0806 10:54:27.065684 (+ 5082us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:27.065690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:27.065712 (+ 22us) server_negotiation.cc:407] Connection header received
0806 10:54:27.065785 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:27.065791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:27.065894 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:27.066295 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:27.068920 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:27.070020 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:27.073316 (+ 3296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:27.073940 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:27.074078 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:28.212012 (+1137934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.2:47643: BlockingRecv error: recv got EOF from 127.30.218.2:47643 (error 108)
Metrics: {"server-negotiator.queue_time_us":4914,"thread_start_us":4734,"threads_started":1}
I20260806 10:54:28.213287 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:27.061008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.2:47643)
0806 10:54:27.062020 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:27.062069 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:27.063973 (+ 1904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:27.066503 (+ 2530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:27.066520 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:27.067507 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:27.068612 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:27.068647 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:27.070225 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:27.070232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:27.072654 (+ 2422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:27.072679 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:27.074291 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:27.074300 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:27.074805 (+ 505us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:27.075787 (+ 982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:27.075826 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:28.211821 (+1135995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":400,"threads_started":1}
W20260806 10:54:28.214325 8449 heartbeater.cc:644] Failed to heartbeat to 127.30.218.62:39573 (0 consecutive failures): Not authorized: Failed to ping master at 127.30.218.62:39573: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Aug 06 10:54:28 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:28 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:28.231221 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:28.215047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.2:42371)
0806 10:54:28.215404 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:28.215432 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:28.215593 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:28.215997 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:28.216002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:28.216575 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:28.216854 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:28.216869 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:28.218069 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:28.218076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:28.219855 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:28.219880 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:28.220964 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:28.220972 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:28.221198 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:28.226153 (+ 4955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:28.226180 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:28.230823 (+ 4643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157}
W20260806 10:54:28.231277 8451 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:28.215187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:42371 (local address 127.30.218.62:39573)
0806 10:54:28.215461 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:28.215471 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:28.215498 (+ 27us) server_negotiation.cc:407] Connection header received
0806 10:54:28.215769 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:28.215774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:28.215848 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:28.215935 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:28.217043 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:28.217923 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:28.220107 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:28.220663 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:28.220792 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:28.231093 (+ 10301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.2:42371: BlockingRecv error: recv got EOF from 127.30.218.2:42371 (error 108)
Metrics: {"server-negotiator.queue_time_us":160}
Aug 06 10:54:28 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:28 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:28.243598 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:28.232693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.2:53887)
0806 10:54:28.233022 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:28.233044 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:28.233180 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:28.233747 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:28.233752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:28.234240 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:28.234436 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:28.234450 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:28.235859 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:28.235865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:28.237009 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:28.237029 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:28.238187 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:28.238193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:28.238346 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:28.239017 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:28.239038 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:28.243367 (+ 4329us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182}
W20260806 10:54:28.243983 8449 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 10:54:28.244438 8451 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:28.233187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:53887 (local address 127.30.218.62:39573)
0806 10:54:28.233389 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:28.233393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:28.233413 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:54:28.233507 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:28.233512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:28.233594 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:28.233691 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:28.234619 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:28.235634 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:28.237214 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:28.237997 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:28.238156 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:28.244075 (+ 5919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.2:53887: BlockingRecv error: recv got EOF from 127.30.218.2:53887 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 06 10:54:28 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:28 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:28.609227 8590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:28.598548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.1:38211)
0806 10:54:28.599112 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:28.599133 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:28.599257 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:28.599699 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:28.599703 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:28.600155 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:28.600320 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:28.600330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:28.601939 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:28.601945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:28.603216 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:28.603233 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:28.604198 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:28.604205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:28.604366 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:28.604955 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:28.604973 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:28.608893 (+ 3920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":317,"threads_started":1}
W20260806 10:54:28.609406 8451 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:28.598762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:38211 (local address 127.30.218.62:39573)
0806 10:54:28.599002 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:28.599010 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:28.599228 (+ 218us) server_negotiation.cc:407] Connection header received
0806 10:54:28.599383 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:28.599390 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:28.599512 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:28.599665 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:28.600474 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:28.601745 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:28.603429 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:28.604001 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:28.604127 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:28.609181 (+ 5054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.1:38211: BlockingRecv error: recv got EOF from 127.30.218.1:38211 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260806 10:54:28.857700 8588 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.015s user 0.005s sys 0.011s
W20260806 10:54:28.859184 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:27.825540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:40513 (local address 127.30.218.62:39573)
0806 10:54:27.827700 (+ 2160us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:27.827706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:27.831554 (+ 3848us) server_negotiation.cc:407] Connection header received
0806 10:54:27.834532 (+ 2978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:27.834538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:27.834641 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:27.834759 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:27.837942 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:27.838972 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:27.841279 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:27.841823 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:27.841962 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:28.858873 (+1016911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.3:40513: BlockingRecv error: recv got EOF from 127.30.218.3:40513 (error 108)
Metrics: {"server-negotiator.queue_time_us":2012,"thread_start_us":1852,"threads_started":1}
I20260806 10:54:28.859781 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:27.825885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.3:40513)
0806 10:54:27.830996 (+ 5111us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:27.831079 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:27.834122 (+ 3043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:27.835147 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:27.835163 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:27.836179 (+ 1016us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:27.837624 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:27.837672 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:27.839200 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:27.839205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:27.841031 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:27.841056 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:27.842123 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:27.842132 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:27.842410 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:27.843080 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:27.843108 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:28.858640 (+1015532us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4688,"thread_start_us":4401,"threads_started":1}
W20260806 10:54:28.860772 8584 heartbeater.cc:644] Failed to heartbeat to 127.30.218.62:39573 (0 consecutive failures): Not authorized: Failed to ping master at 127.30.218.62:39573: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Aug 06 10:54:28 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:28 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:28.870585 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:28.861423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.3:49533)
0806 10:54:28.861642 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:28.861659 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:28.861784 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:28.862148 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:28.862154 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:28.862560 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:28.862759 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:28.862770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:28.863845 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:28.863850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:28.864854 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:28.864866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:28.865683 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:28.865690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:28.865850 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:28.866494 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:28.866512 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:28.870268 (+ 3756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":80}
W20260806 10:54:28.870769 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:28.861571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:49533 (local address 127.30.218.62:39573)
0806 10:54:28.861863 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:28.861868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:28.861886 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:54:28.861941 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:28.861945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:28.862004 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:28.862094 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:28.862915 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:28.863686 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:28.865015 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:28.865438 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:28.865599 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:28.870527 (+ 4928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.3:49533: BlockingRecv error: recv got EOF from 127.30.218.3:49533 (error 108)
Metrics: {"server-negotiator.queue_time_us":159}
Aug 06 10:54:28 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:28 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:28.880178 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:28.871500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.3:33207)
0806 10:54:28.871671 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:28.871693 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:28.871805 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:28.872136 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:28.872139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:28.872485 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:28.872680 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:28.872691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:28.873818 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:28.873824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:28.874826 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:28.874838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:28.875546 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:28.875550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:28.875682 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:28.876259 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:28.876276 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:28.879880 (+ 3604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 10:54:28.880436 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:28.871623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:33207 (local address 127.30.218.62:39573)
0806 10:54:28.871784 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:28.871789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:28.871808 (+ 19us) server_negotiation.cc:407] Connection header received
0806 10:54:28.871933 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:28.871939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:28.872008 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:28.872088 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:28.872839 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:28.873681 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:28.874950 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:28.875416 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:28.875512 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:28.880165 (+ 4653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.3:33207: BlockingRecv error: recv got EOF from 127.30.218.3:33207 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 10:54:28.880554 8584 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 10:54:29 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:29 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:29.254388 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:29.244672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.2:33807)
0806 10:54:29.245294 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:29.245312 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:29.245484 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:29.245857 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:29.245862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:29.246349 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:29.246517 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:29.246528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:29.247584 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:29.247588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:29.248628 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:29.248642 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:29.249400 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:29.249407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:29.249634 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:29.250370 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:29.250389 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:29.254128 (+ 3739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":369,"threads_started":1}
W20260806 10:54:29.254560 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:29.244838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:33807 (local address 127.30.218.62:39573)
0806 10:54:29.245100 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:29.245107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:29.245409 (+ 302us) server_negotiation.cc:407] Connection header received
0806 10:54:29.245619 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:29.245623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:29.245708 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:29.245798 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:29.246691 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:29.247425 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:29.248773 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:29.249209 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:29.249329 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:29.254366 (+ 5037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.2:33807: BlockingRecv error: recv got EOF from 127.30.218.2:33807 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Aug 06 10:54:29 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:29 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:29.620400 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:29.610562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.1:50169)
0806 10:54:29.611115 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:29.611132 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:29.611276 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:29.611628 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:29.611632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:29.612088 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:29.612286 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:29.612296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:29.613413 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:29.613418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:29.614413 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:29.614426 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:29.615244 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:29.615250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:29.615423 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:29.616034 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:29.616053 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:29.620112 (+ 4059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":310,"threads_started":1}
W20260806 10:54:29.620543 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:29.610713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:50169 (local address 127.30.218.62:39573)
0806 10:54:29.610959 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:29.610966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:29.611231 (+ 265us) server_negotiation.cc:407] Connection header received
0806 10:54:29.611407 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:29.611413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:29.611478 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:29.611568 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:29.612458 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:29.613244 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:29.614581 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:29.615067 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:29.615183 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:29.620321 (+ 5138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.1:50169: BlockingRecv error: recv got EOF from 127.30.218.1:50169 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Aug 06 10:54:29 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:29 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:29.891253 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:29.881373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.3:55619)
0806 10:54:29.881991 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:29.882014 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:29.882170 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:29.882618 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:29.882624 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:29.883198 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:29.883365 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:29.883376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:29.884485 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:29.884489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:29.885549 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:29.885564 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:29.886405 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:29.886411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:29.886583 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:29.887201 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:29.887219 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:29.890882 (+ 3663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":358,"threads_started":1}
W20260806 10:54:29.891445 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:29.881602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:55619 (local address 127.30.218.62:39573)
0806 10:54:29.881790 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:29.881797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:29.882124 (+ 327us) server_negotiation.cc:407] Connection header received
0806 10:54:29.882297 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:29.882303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:29.882415 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:29.882517 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:29.883551 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:29.884320 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:29.885714 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:29.886202 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:29.886325 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:29.891201 (+ 4876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.3:55619: BlockingRecv error: recv got EOF from 127.30.218.3:55619 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 10:54:30 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:30 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:30.265502 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:30.255743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.2:51351)
0806 10:54:30.256351 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:30.256370 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:30.256544 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:30.256914 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:30.256919 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:30.257394 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:30.257659 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:30.257671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:30.258714 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:30.258718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:30.259656 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:30.259669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:30.260437 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:30.260444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:30.260631 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:30.261318 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:30.261337 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:30.265150 (+ 3813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":340,"threads_started":1}
W20260806 10:54:30.265749 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:30.255918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:51351 (local address 127.30.218.62:39573)
0806 10:54:30.256211 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:30.256217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:30.256473 (+ 256us) server_negotiation.cc:407] Connection header received
0806 10:54:30.256683 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:30.256688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:30.256759 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:30.256847 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:30.257819 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:30.258552 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:30.259808 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:30.260259 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:30.260376 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:30.265468 (+ 5092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.2:51351: BlockingRecv error: recv got EOF from 127.30.218.2:51351 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Aug 06 10:54:30 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:30 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:30.630887 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:30.621593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.1:40193)
0806 10:54:30.622116 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:30.622134 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:30.622274 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:30.622620 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:30.622623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:30.623005 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:30.623172 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:30.623195 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:30.624290 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:30.624293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:30.625320 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:30.625334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:30.626267 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:30.626274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:30.626412 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:30.627044 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:30.627063 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:30.630632 (+ 3569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":282,"threads_started":1}
W20260806 10:54:30.631064 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:30.621825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:40193 (local address 127.30.218.62:39573)
0806 10:54:30.622020 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:30.622024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:30.622203 (+ 179us) server_negotiation.cc:407] Connection header received
0806 10:54:30.622420 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:30.622424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:30.622495 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:30.622589 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:30.623356 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:30.624167 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:30.625488 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:30.626101 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:30.626203 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:30.630858 (+ 4655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.1:40193: BlockingRecv error: recv got EOF from 127.30.218.1:40193 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 10:54:30 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:30 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:30.901625 8596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:30.892368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.3:45541)
0806 10:54:30.892937 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:30.892961 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:30.893078 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:30.893380 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:30.893383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:30.893815 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:30.893997 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:30.894005 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:30.895378 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:30.895383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:30.896300 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:30.896312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:30.896925 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:30.896929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:30.897048 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:30.897675 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:30.897700 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:30.901322 (+ 3622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":345,"threads_started":1}
W20260806 10:54:30.901824 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:30.892478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:45541 (local address 127.30.218.62:39573)
0806 10:54:30.892658 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:30.892662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:30.893029 (+ 367us) server_negotiation.cc:407] Connection header received
0806 10:54:30.893181 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:30.893185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:30.893257 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:30.893342 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:30.894131 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:30.895238 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:30.896408 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:30.896810 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:30.896907 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:30.901589 (+ 4682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.3:45541: BlockingRecv error: recv got EOF from 127.30.218.3:45541 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 06 10:54:31 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:31 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:31.276135 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:31.266660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.2:37395)
0806 10:54:31.267194 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:31.267211 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:31.267395 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:31.267729 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:31.267732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:31.268108 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:31.268324 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:31.268338 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:31.269524 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:31.269528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:31.270487 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:31.270500 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:31.271210 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:31.271214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:31.271336 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:31.271951 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:31.271968 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:31.275821 (+ 3853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":284,"threads_started":1}
W20260806 10:54:31.276333 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:31.266792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:37395 (local address 127.30.218.62:39573)
0806 10:54:31.266963 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:31.266968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:31.267323 (+ 355us) server_negotiation.cc:407] Connection header received
0806 10:54:31.267518 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:31.267524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:31.267595 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:31.267685 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:31.268491 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:31.269350 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:31.270640 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:31.271083 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:31.271183 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:31.276092 (+ 4909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.2:37395: BlockingRecv error: recv got EOF from 127.30.218.2:37395 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 10:54:31 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:31 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:31.641394 8598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:31.631986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.1:41823)
0806 10:54:31.632495 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:31.632510 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:31.632647 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:31.632975 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:31.632979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:31.633395 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:31.633632 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:31.633643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:31.634671 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:31.634675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:31.635668 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:31.635680 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:31.636417 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:31.636423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:31.636584 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:31.637208 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:31.637231 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:31.641039 (+ 3808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":269,"threads_started":1}
W20260806 10:54:31.641592 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:31.632111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:41823 (local address 127.30.218.62:39573)
0806 10:54:31.632334 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:31.632340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:31.632592 (+ 252us) server_negotiation.cc:407] Connection header received
0806 10:54:31.632775 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:31.632780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:31.632842 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:31.632924 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:31.633783 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:31.634517 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:31.635830 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:31.636268 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:31.636380 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:31.641319 (+ 4939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.1:41823: BlockingRecv error: recv got EOF from 127.30.218.1:41823 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 06 10:54:31 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:31 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:31.911159 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:31.902673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.3:55615)
0806 10:54:31.903192 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:31.903219 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:31.903364 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:31.903748 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:31.903751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:31.904120 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:31.904302 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:31.904311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:31.905290 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:31.905294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:31.906227 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:31.906237 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:31.906843 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:31.906847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:31.906964 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:31.907535 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:31.907553 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:31.910922 (+ 3369us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":269,"threads_started":1}
W20260806 10:54:31.911355 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:31.902702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:55615 (local address 127.30.218.62:39573)
0806 10:54:31.902919 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:31.902925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:31.903292 (+ 367us) server_negotiation.cc:407] Connection header received
0806 10:54:31.903512 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:31.903517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:31.903619 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:31.903699 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:31.904447 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:31.905187 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:31.906336 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:31.906736 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:31.906824 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:31.911148 (+ 4324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.3:55615: BlockingRecv error: recv got EOF from 127.30.218.3:55615 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 06 10:54:32 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:32 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:32.288020 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:32.277315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.2:58355)
0806 10:54:32.277936 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:32.277955 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:32.278112 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:32.278536 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:32.278541 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:32.279047 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:32.279248 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:32.279259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:32.280508 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:32.280512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:32.282031 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:32.282046 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:32.282850 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:32.282854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:32.282987 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:32.283705 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:32.283725 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:32.287688 (+ 3963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":349,"threads_started":1}
W20260806 10:54:32.288206 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:32.277379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:58355 (local address 127.30.218.62:39573)
0806 10:54:32.277624 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:32.277631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:32.278060 (+ 429us) server_negotiation.cc:407] Connection header received
0806 10:54:32.278245 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:32.278250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:32.278362 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:32.278463 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:32.279408 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:32.280348 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:32.282178 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:32.282683 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:32.282793 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:32.287942 (+ 5149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.2:58355: BlockingRecv error: recv got EOF from 127.30.218.2:58355 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 06 10:54:32 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:32 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:32.653294 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:32.642604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.1:34649)
0806 10:54:32.643196 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:32.643216 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:32.643383 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:32.643713 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:32.643717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:32.644281 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:32.644484 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:32.644497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:32.645626 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:32.645631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:32.646891 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:32.646915 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:32.647789 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:32.647796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:32.648012 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:32.648829 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:32.648852 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:32.652993 (+ 4141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":318,"threads_started":1}
W20260806 10:54:32.653488 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:32.642701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:34649 (local address 127.30.218.62:39573)
0806 10:54:32.642953 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:32.642961 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:32.643318 (+ 357us) server_negotiation.cc:407] Connection header received
0806 10:54:32.643501 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:32.643505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:32.643579 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:32.643666 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:32.644645 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:32.645494 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:32.647054 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:32.647592 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:32.647727 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:32.653224 (+ 5497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.1:34649: BlockingRecv error: recv got EOF from 127.30.218.1:34649 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 06 10:54:32 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:32 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:32.927549 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:32.912321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.3:56627)
0806 10:54:32.912961 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:32.912986 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:32.913122 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:32.913595 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:32.913600 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:32.914056 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:32.914261 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:32.914272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:32.915777 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:32.915783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:32.916921 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:32.916937 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:32.917762 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:32.917768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:32.917923 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:32.922693 (+ 4770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:32.922718 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:32.927089 (+ 4371us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":341,"threads_started":1}
W20260806 10:54:32.927775 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:32.912485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:56627 (local address 127.30.218.62:39573)
0806 10:54:32.912697 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:32.912702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:32.913096 (+ 394us) server_negotiation.cc:407] Connection header received
0806 10:54:32.913237 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:32.913242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:32.913394 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:32.913537 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:32.914470 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:32.915624 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:32.917068 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:32.917591 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:32.917703 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:32.927413 (+ 9710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.3:56627: BlockingRecv error: recv got EOF from 127.30.218.3:56627 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 06 10:54:33 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:33 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:33.304446 8603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:33.289217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.2:40281)
0806 10:54:33.289850 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:33.289869 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:33.290002 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:33.290498 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:33.290503 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:33.291076 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:33.291262 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:33.291273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:33.292339 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:33.292344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:33.293358 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:33.293372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:33.294135 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:33.294140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:33.294282 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:33.299780 (+ 5498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:33.299818 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:33.304172 (+ 4354us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":317,"threads_started":1}
W20260806 10:54:33.304726 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:33.289406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:40281 (local address 127.30.218.62:39573)
0806 10:54:33.289647 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:33.289653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:33.289973 (+ 320us) server_negotiation.cc:407] Connection header received
0806 10:54:33.290208 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:33.290216 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:33.290313 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:33.290441 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:33.291411 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:33.292186 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:33.293531 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:33.293958 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:33.294073 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:33.304426 (+ 10353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.2:40281: BlockingRecv error: recv got EOF from 127.30.218.2:40281 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 06 10:54:33 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:33 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:33.664384 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:33.654506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.1:41091)
0806 10:54:33.655081 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:33.655100 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:33.655275 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:33.655643 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:33.655647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:33.656275 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:33.656532 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:33.656551 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:33.657587 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:33.657590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:33.658605 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:33.658620 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:33.659437 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:33.659442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:33.659590 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:33.660265 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:33.660285 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:33.664095 (+ 3810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":285,"threads_started":1}
W20260806 10:54:33.664587 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:33.654878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:41091 (local address 127.30.218.62:39573)
0806 10:54:33.655078 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:33.655083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:33.655203 (+ 120us) server_negotiation.cc:407] Connection header received
0806 10:54:33.655416 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:33.655421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:33.655503 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:33.655623 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:33.656682 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:33.657479 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:33.658774 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:33.659281 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:33.659389 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:33.664345 (+ 4956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.1:41091: BlockingRecv error: recv got EOF from 127.30.218.1:41091 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 06 10:54:33 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:33 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:33.939857 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:33.928882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.3:44049)
0806 10:54:33.929507 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:33.929535 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:33.929705 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:33.930054 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:33.930058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:33.930668 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:33.930858 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:33.930869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:33.932259 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:33.932264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:33.933244 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:33.933257 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:33.933948 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:33.933952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:33.934081 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:33.934735 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:33.934763 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:33.939484 (+ 4721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":402,"threads_started":1}
W20260806 10:54:33.940094 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:33.929050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:44049 (local address 127.30.218.62:39573)
0806 10:54:33.929290 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:33.929295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:33.929643 (+ 348us) server_negotiation.cc:407] Connection header received
0806 10:54:33.929836 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:33.929841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:33.929909 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:33.929995 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:33.931005 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:33.932096 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:33.933384 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:33.933831 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:33.933936 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:33.939831 (+ 5895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.3:44049: BlockingRecv error: recv got EOF from 127.30.218.3:44049 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 06 10:54:34 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:34 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:34.315225 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:34.305578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.2:54791)
0806 10:54:34.306055 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:34.306072 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:34.306198 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:34.306500 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:34.306503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:34.306947 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:34.307140 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:34.307150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:34.308176 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:34.308181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:34.309157 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:34.309169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:34.309979 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:34.309984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:34.310127 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:34.310752 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:34.310770 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:34.314876 (+ 4106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":242,"threads_started":1}
W20260806 10:54:34.315400 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:34.305679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:54791 (local address 127.30.218.62:39573)
0806 10:54:34.305990 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:34.305996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:34.306150 (+ 154us) server_negotiation.cc:407] Connection header received
0806 10:54:34.306296 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:34.306301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:34.306369 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:34.306452 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:34.307267 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:34.308015 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:34.309316 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:34.309812 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:34.309913 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:34.315142 (+ 5229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.2:54791: BlockingRecv error: recv got EOF from 127.30.218.2:54791 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Aug 06 10:54:34 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:34 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:34.675022 8607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:34.665540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.1:34759)
0806 10:54:34.666146 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:34.666163 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:34.666284 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:34.666560 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:34.666563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:34.666981 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:34.667182 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:34.667192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:34.668255 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:34.668259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:34.669321 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:34.669336 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:34.670126 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:34.670131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:34.670263 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:34.670888 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:34.670907 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:34.674676 (+ 3769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":262,"threads_started":1}
W20260806 10:54:34.675105 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:34.665628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:34759 (local address 127.30.218.62:39573)
0806 10:54:34.665840 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:34.665848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:34.666242 (+ 394us) server_negotiation.cc:407] Connection header received
0806 10:54:34.666379 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:34.666383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:34.666448 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:34.666531 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:34.667306 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:34.668096 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:34.669504 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:34.669960 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:34.670066 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:34.674909 (+ 4843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.1:34759: BlockingRecv error: recv got EOF from 127.30.218.1:34759 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 06 10:54:34 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:34 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:34.950450 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:34.941112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.3:48853)
0806 10:54:34.941605 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:34.941628 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:34.941750 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:34.942029 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:34.942032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:34.942477 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:34.942652 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:34.942661 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:34.943719 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:34.943723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:34.944661 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:34.944672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:34.945472 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:34.945478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:34.945620 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:34.946234 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:34.946252 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:34.950124 (+ 3872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":238,"threads_started":1}
W20260806 10:54:34.950691 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:34.941246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:48853 (local address 127.30.218.62:39573)
0806 10:54:34.941511 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:34.941517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:34.941700 (+ 183us) server_negotiation.cc:407] Connection header received
0806 10:54:34.941849 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:34.941853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:34.941919 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:34.941990 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:34.942817 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:34.943592 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:34.944824 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:34.945307 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:34.945415 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:34.950436 (+ 5021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.3:48853: BlockingRecv error: recv got EOF from 127.30.218.3:48853 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 06 10:54:35 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:35 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:35.325898 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:35.316213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.2:54827)
0806 10:54:35.316775 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:35.316792 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:35.316936 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:35.317274 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:35.317278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:35.317777 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:35.317968 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:35.317979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:35.319063 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:35.319070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:35.320078 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:35.320093 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:35.320878 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:35.320884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:35.321057 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:35.321753 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:35.321772 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:35.325609 (+ 3837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":264,"threads_started":1}
W20260806 10:54:35.326094 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:35.316287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:54827 (local address 127.30.218.62:39573)
0806 10:54:35.316538 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:35.316544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:35.316885 (+ 341us) server_negotiation.cc:407] Connection header received
0806 10:54:35.317063 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:35.317067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:35.317134 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:35.317217 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:35.318127 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:35.318910 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:35.320240 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:35.320713 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:35.320827 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:35.325884 (+ 5057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.2:54827: BlockingRecv error: recv got EOF from 127.30.218.2:54827 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Aug 06 10:54:35 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:35 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:35.686012 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:35.676321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.1:58087)
0806 10:54:35.676846 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:35.676864 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:35.676994 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:35.677298 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:35.677301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:35.677769 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:35.677946 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:35.677957 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:35.679065 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:35.679070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:35.680363 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:35.680381 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:35.681325 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:35.681338 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:35.681527 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:35.682182 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:35.682202 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:35.685742 (+ 3540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":278,"threads_started":1}
W20260806 10:54:35.686337 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:35.676465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:58087 (local address 127.30.218.62:39573)
0806 10:54:35.676711 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:35.676717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:35.676946 (+ 229us) server_negotiation.cc:407] Connection header received
0806 10:54:35.677093 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:35.677097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:35.677176 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:35.677262 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:35.678104 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:35.678891 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:35.680509 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:35.681126 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:35.681252 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:35.686041 (+ 4789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.1:58087: BlockingRecv error: recv got EOF from 127.30.218.1:58087 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 06 10:54:35 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:35 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:35.961517 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:35.951704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.3:46737)
0806 10:54:35.952598 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:35.952623 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:35.952784 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:35.953173 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:35.953177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:35.953676 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:35.953867 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:35.953879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:35.954939 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:35.954943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:35.955938 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:35.955952 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:35.956773 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:35.956781 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:35.956946 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:35.957570 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:35.957596 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:35.961208 (+ 3612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":785,"thread_start_us":540,"threads_started":1}
W20260806 10:54:35.961694 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:35.951891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:46737 (local address 127.30.218.62:39573)
0806 10:54:35.952159 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:35.952167 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:35.952720 (+ 553us) server_negotiation.cc:407] Connection header received
0806 10:54:35.952919 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:35.952925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:35.953029 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:35.953130 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:35.954026 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:35.954787 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:35.956092 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:35.956597 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:35.956731 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:35.961419 (+ 4688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.3:46737: BlockingRecv error: recv got EOF from 127.30.218.3:46737 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 06 10:54:36 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:36 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:36.336906 8612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:36.327236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.2:50257)
0806 10:54:36.327821 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:36.327839 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:36.327966 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:36.328298 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:36.328302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:36.328725 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:36.328908 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:36.328917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:36.330509 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:36.330513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:36.331492 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:36.331506 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:36.332349 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:36.332355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:36.332513 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:36.333157 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:36.333177 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:36.336567 (+ 3390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":319,"threads_started":1}
W20260806 10:54:36.337095 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:36.327368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:50257 (local address 127.30.218.62:39573)
0806 10:54:36.327543 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:36.327547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:36.327913 (+ 366us) server_negotiation.cc:407] Connection header received
0806 10:54:36.328065 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:36.328069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:36.328137 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:36.328233 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:36.329025 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:36.330327 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:36.331665 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:36.332143 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:36.332263 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:36.336842 (+ 4579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.2:50257: BlockingRecv error: recv got EOF from 127.30.218.2:50257 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 06 10:54:36 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:36 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:36.698415 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:36.687246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.1:49339)
0806 10:54:36.687775 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:36.687793 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:36.687926 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:36.688375 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:36.688379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:36.688799 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:36.688956 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:36.688966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:36.690171 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:36.690176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:36.691365 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:36.691382 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:36.692400 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:36.692407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:36.692584 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:36.693582 (+ 998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:36.693633 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:36.698076 (+ 4443us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":263,"threads_started":1}
W20260806 10:54:36.698625 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:36.687382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:49339 (local address 127.30.218.62:39573)
0806 10:54:36.687610 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:36.687614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:36.687867 (+ 253us) server_negotiation.cc:407] Connection header received
0806 10:54:36.688110 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:36.688115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:36.688188 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:36.688282 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:36.689128 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:36.689984 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:36.691584 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:36.692210 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:36.692339 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:36.698378 (+ 6039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.1:49339: BlockingRecv error: recv got EOF from 127.30.218.1:49339 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 06 10:54:36 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:36 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:36.972827 8614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:36.962584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.3:36231)
0806 10:54:36.963157 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:36.963182 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:36.963328 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:36.963655 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:36.963658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:36.964104 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:36.964289 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:36.964299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:36.965615 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:36.965621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:36.966891 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:36.966910 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:36.967679 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:36.967684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:36.967852 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:36.968587 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:36.968608 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:36.972421 (+ 3813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":293,"threads_started":1}
W20260806 10:54:36.973026 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:36.962863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:36231 (local address 127.30.218.62:39573)
0806 10:54:36.963079 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:36.963084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:36.963327 (+ 243us) server_negotiation.cc:407] Connection header received
0806 10:54:36.963430 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:36.963434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:36.963520 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:36.963632 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:36.964468 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:36.965409 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:36.967030 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:36.967497 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:36.967606 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:36.972765 (+ 5159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.3:36231: BlockingRecv error: recv got EOF from 127.30.218.3:36231 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 06 10:54:37 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:37 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:37.348529 8615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:37.338141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.2:41869)
0806 10:54:37.338841 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:37.338860 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:37.339008 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:37.339378 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:37.339382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:37.339894 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:37.340100 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:37.340113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:37.341223 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:37.341228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:37.342336 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:37.342351 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:37.343055 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:37.343060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:37.343213 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:37.343898 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:37.343917 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:37.348119 (+ 4202us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":386,"threads_started":1}
W20260806 10:54:37.348630 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:37.338168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:41869 (local address 127.30.218.62:39573)
0806 10:54:37.338388 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:37.338393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:37.338943 (+ 550us) server_negotiation.cc:407] Connection header received
0806 10:54:37.339124 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:37.339129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:37.339204 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:37.339286 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:37.340250 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:37.341070 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:37.342456 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:37.342922 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:37.343013 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:37.348379 (+ 5366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.2:41869: BlockingRecv error: recv got EOF from 127.30.218.2:41869 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 06 10:54:37 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:37 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:37.709722 8616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:37.699513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.1:55311)
0806 10:54:37.700102 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:37.700120 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:37.700259 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:37.700690 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:37.700694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:37.701174 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:37.701384 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:37.701396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:37.702645 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:37.702650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:37.703753 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:37.703768 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:37.704531 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:37.704535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:37.704669 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:37.705390 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:37.705411 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:37.709329 (+ 3918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":324,"threads_started":1}
W20260806 10:54:37.709913 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:37.699583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:55311 (local address 127.30.218.62:39573)
0806 10:54:37.699810 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:37.699816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:37.700203 (+ 387us) server_negotiation.cc:407] Connection header received
0806 10:54:37.700419 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:37.700424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:37.700502 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:37.700588 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:37.701581 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:37.702499 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:37.703899 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:37.704386 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:37.704485 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:37.709639 (+ 5154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.1:55311: BlockingRecv error: recv got EOF from 127.30.218.1:55311 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 06 10:54:37 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:37 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:37.984268 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:37.974175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.3:53679)
0806 10:54:37.974838 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:37.974862 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:37.975038 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:37.975416 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:37.975421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:37.975929 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:37.976112 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:37.976122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:37.977253 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:37.977259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:37.978314 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:37.978328 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:37.979087 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:37.979091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:37.979274 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:37.980067 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:37.980093 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:37.983987 (+ 3894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":363,"threads_started":1}
W20260806 10:54:37.984431 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:37.974254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:53679 (local address 127.30.218.62:39573)
0806 10:54:37.974533 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:37.974540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:37.974961 (+ 421us) server_negotiation.cc:407] Connection header received
0806 10:54:37.975171 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:37.975178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:37.975262 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:37.975389 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:37.976262 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:37.977110 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:37.978456 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:37.978928 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:37.979058 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:37.984201 (+ 5143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.3:53679: BlockingRecv error: recv got EOF from 127.30.218.3:53679 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 06 10:54:38 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:38 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:38.362485 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:38.349676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.2:50719)
0806 10:54:38.350247 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:38.350264 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:38.350451 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:38.350800 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:38.350805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:38.351312 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:38.351512 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:38.351544 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:38.352676 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:38.352682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:38.353844 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:38.353859 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:38.354607 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:38.354613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:38.354775 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:38.358045 (+ 3270us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:38.358066 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:38.362183 (+ 4117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":290,"threads_started":1}
W20260806 10:54:38.362850 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:38.349742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:50719 (local address 127.30.218.62:39573)
0806 10:54:38.349963 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:38.349970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:38.350383 (+ 413us) server_negotiation.cc:407] Connection header received
0806 10:54:38.350574 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:38.350581 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:38.350662 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:38.350747 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:38.351702 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:38.352522 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:38.353980 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:38.354458 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:38.354571 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:38.362516 (+ 7945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.2:50719: BlockingRecv error: recv got EOF from 127.30.218.2:50719 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 10:54:38 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:38 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:38.721264 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:38.711003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.1:48971)
0806 10:54:38.711621 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:38.711639 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:38.711796 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:38.712166 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:38.712172 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:38.712735 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:38.713006 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:38.713023 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:38.714396 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:38.714404 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:38.715583 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:38.715599 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:38.716328 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:38.716335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:38.716474 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:38.717130 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:38.717148 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:38.721006 (+ 3858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":367,"threads_started":1}
W20260806 10:54:38.721675 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:38.711220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:48971 (local address 127.30.218.62:39573)
0806 10:54:38.711461 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:38.711468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:38.711724 (+ 256us) server_negotiation.cc:407] Connection header received
0806 10:54:38.711941 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:38.711946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:38.712018 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:38.712105 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:38.713177 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:38.714242 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:38.715729 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:38.716188 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:38.716286 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:38.721227 (+ 4941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.1:48971: BlockingRecv error: recv got EOF from 127.30.218.1:48971 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 06 10:54:38 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:38 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:38.996449 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:38.985384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.3:59247)
0806 10:54:38.986948 (+ 1564us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:38.986971 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:38.987139 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:38.987494 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:38.987499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:38.987982 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:38.988174 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:38.988184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:38.989295 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:38.989301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:38.990421 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:38.990434 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:38.991190 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:38.991195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:38.991375 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:38.992014 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:38.992033 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:38.996083 (+ 4050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1443,"thread_start_us":1258,"threads_started":1}
W20260806 10:54:38.996622 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:38.985512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:59247 (local address 127.30.218.62:39573)
0806 10:54:38.985762 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:38.985769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:38.987066 (+ 1297us) server_negotiation.cc:407] Connection header received
0806 10:54:38.987264 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:38.987271 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:38.987357 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:38.987442 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:38.988325 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:38.989149 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:38.990556 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:38.991038 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:38.991149 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:38.996322 (+ 5173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.3:59247: BlockingRecv error: recv got EOF from 127.30.218.3:59247 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 06 10:54:39 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:39 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:39.373510 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:39.363671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.2:38495)
0806 10:54:39.364196 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:39.364214 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:39.364355 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:39.364682 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:39.364686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:39.365219 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:39.365496 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:39.365513 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:39.366898 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:39.366904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:39.367876 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:39.367890 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:39.368645 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:39.368649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:39.368790 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:39.369435 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:39.369490 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:39.373225 (+ 3735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":266,"threads_started":1}
W20260806 10:54:39.373764 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:39.363799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:38495 (local address 127.30.218.62:39573)
0806 10:54:39.364104 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:39.364111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:39.364291 (+ 180us) server_negotiation.cc:407] Connection header received
0806 10:54:39.364482 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:39.364487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:39.364567 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:39.364646 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:39.365641 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:39.366776 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:39.368023 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:39.368511 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:39.368631 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:39.373480 (+ 4849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.2:38495: BlockingRecv error: recv got EOF from 127.30.218.2:38495 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Aug 06 10:54:39 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:39 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:39.732648 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:39.722569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.1:49747)
0806 10:54:39.723286 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:39.723305 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:39.723451 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:39.723917 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:39.723921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:39.724434 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:39.724640 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:39.724652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:39.725860 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:39.725867 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:39.727249 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:39.727267 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:39.727973 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:39.727978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:39.728112 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:39.728710 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:39.728738 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:39.732370 (+ 3632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":394,"threads_started":1}
W20260806 10:54:39.732844 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:39.722712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:49747 (local address 127.30.218.62:39573)
0806 10:54:39.722952 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:39.722956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:39.723402 (+ 446us) server_negotiation.cc:407] Connection header received
0806 10:54:39.723610 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:39.723616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:39.723691 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:39.723793 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:39.724802 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:39.725703 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:39.727396 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:39.727821 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:39.727914 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:39.732620 (+ 4706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.1:49747: BlockingRecv error: recv got EOF from 127.30.218.1:49747 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Aug 06 10:54:40 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:40 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:40.008112 8624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:39.997776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.3:48129)
0806 10:54:39.998517 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:39.998549 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:39.998752 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:39.999347 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:39.999351 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:39.999940 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:40.000188 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:40.000204 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:40.001651 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:40.001657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:40.002616 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:40.002628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:40.003373 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:40.003379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:40.003566 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:40.004137 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:40.004155 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:40.007792 (+ 3637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":498,"threads_started":1}
W20260806 10:54:40.008246 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:39.997896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:48129 (local address 127.30.218.62:39573)
0806 10:54:39.998120 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:39.998127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:39.998631 (+ 504us) server_negotiation.cc:407] Connection header received
0806 10:54:39.999085 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:39.999091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:39.999176 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:39.999284 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:40.000336 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:40.001503 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:40.002760 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:40.003214 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:40.003323 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:40.008023 (+ 4700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.3:48129: BlockingRecv error: recv got EOF from 127.30.218.3:48129 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 06 10:54:40 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:40 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:40.382632 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:40.374420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.2:45595)
0806 10:54:40.374962 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:40.374978 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:40.375093 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:40.375372 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:40.375375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:40.375773 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:40.375950 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:40.375959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:40.376914 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:40.376917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:40.377850 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:40.377862 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:40.378537 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:40.378542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:40.378663 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:40.379220 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:40.379237 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:40.382357 (+ 3120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":279,"threads_started":1}
W20260806 10:54:40.382879 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:40.374484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:45595 (local address 127.30.218.62:39573)
0806 10:54:40.374652 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:40.374656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:40.375039 (+ 383us) server_negotiation.cc:407] Connection header received
0806 10:54:40.375187 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:40.375191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:40.375251 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:40.375342 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:40.376075 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:40.376805 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:40.377950 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:40.378398 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:40.378486 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:40.382607 (+ 4121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.2:45595: BlockingRecv error: recv got EOF from 127.30.218.2:45595 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 06 10:54:40 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:40 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:40.742952 8626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:40.733746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.1:35685)
0806 10:54:40.734335 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:40.734353 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:40.734475 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:40.734786 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:40.734789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:40.735260 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:40.735425 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:40.735436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:40.736465 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:40.736469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:40.737434 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:40.737504 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:40.738157 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:40.738161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:40.738297 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:40.738924 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:40.738943 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:40.742671 (+ 3728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":355,"threads_started":1}
W20260806 10:54:40.743286 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:40.733901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:35685 (local address 127.30.218.62:39573)
0806 10:54:40.734108 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:40.734114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:40.734425 (+ 311us) server_negotiation.cc:407] Connection header received
0806 10:54:40.734585 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:40.734589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:40.734665 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:40.734750 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:40.735566 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:40.736308 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:40.737631 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:40.738029 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:40.738135 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:40.742970 (+ 4835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.1:35685: BlockingRecv error: recv got EOF from 127.30.218.1:35685 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 06 10:54:41 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:41 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:41.019025 8627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:41.009242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.3:47367)
0806 10:54:41.009842 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:41.009863 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:41.010008 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:41.010364 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:41.010369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:41.010837 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:41.011022 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:41.011033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:41.012154 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:41.012160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:41.013140 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:41.013153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:41.013962 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:41.013968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:41.014150 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:41.014803 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:41.014822 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:41.018670 (+ 3848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":349,"threads_started":1}
W20260806 10:54:41.019198 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:41.009341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:47367 (local address 127.30.218.62:39573)
0806 10:54:41.009569 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:41.009576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:41.009959 (+ 383us) server_negotiation.cc:407] Connection header received
0806 10:54:41.010137 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:41.010141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:41.010208 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:41.010291 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:41.011191 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:41.011983 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:41.013298 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:41.013767 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:41.013887 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:41.018919 (+ 5032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.3:47367: BlockingRecv error: recv got EOF from 127.30.218.3:47367 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 06 10:54:41 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:41 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:41.393630 8628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:41.383884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.2:59495)
0806 10:54:41.384440 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:41.384458 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:41.384606 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:41.384915 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:41.384918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:41.385351 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:41.385562 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:41.385574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:41.386659 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:41.386664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:41.387622 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:41.387640 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:41.388520 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:41.388525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:41.388708 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:41.389361 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:41.389380 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:41.393230 (+ 3850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":303,"threads_started":1}
W20260806 10:54:41.393821 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:41.384163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:59495 (local address 127.30.218.62:39573)
0806 10:54:41.384372 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:41.384377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:41.384542 (+ 165us) server_negotiation.cc:407] Connection header received
0806 10:54:41.384713 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:41.384718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:41.384784 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:41.384877 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:41.385709 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:41.386484 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:41.387859 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:41.388356 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:41.388473 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:41.393599 (+ 5126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.2:59495: BlockingRecv error: recv got EOF from 127.30.218.2:59495 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 06 10:54:41 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:41 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:41.755064 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:41.744549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.1:37921)
0806 10:54:41.745117 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:41.745134 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:41.745282 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:41.745690 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:41.745694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:41.746212 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:41.746447 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:41.746461 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:41.747721 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:41.747727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:41.748779 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:41.748793 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:41.749549 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:41.749555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:41.749750 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:41.750377 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:41.750397 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:41.754769 (+ 4372us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":273,"threads_started":1}
W20260806 10:54:41.755307 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:41.744764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:37921 (local address 127.30.218.62:39573)
0806 10:54:41.745015 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:41.745021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:41.745231 (+ 210us) server_negotiation.cc:407] Connection header received
0806 10:54:41.745407 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:41.745413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:41.745522 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:41.745624 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:41.746605 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:41.747576 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:41.748921 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:41.749367 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:41.749518 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:41.755064 (+ 5546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.1:37921: BlockingRecv error: recv got EOF from 127.30.218.1:37921 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 06 10:54:42 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:42 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.3@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:42.030077 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:42.020338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.3:33345)
0806 10:54:42.020902 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:42.020931 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:42.021090 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:42.021439 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:42.021471 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:42.021939 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:42.022099 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:42.022109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:42.023267 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:42.023272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:42.024427 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:42.024447 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:42.025400 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:42.025407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:42.025638 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:42.026239 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:42.026257 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:42.029824 (+ 3567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":303,"threads_started":1}
W20260806 10:54:42.030359 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:42.020530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:33345 (local address 127.30.218.62:39573)
0806 10:54:42.020913 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:42.020920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:42.021023 (+ 103us) server_negotiation.cc:407] Connection header received
0806 10:54:42.021220 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:42.021224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:42.021293 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:42.021377 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:42.022258 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:42.023025 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:42.024617 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:42.025229 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:42.025344 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:42.030103 (+ 4759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.3:33345: BlockingRecv error: recv got EOF from 127.30.218.3:33345 (error 108)
Metrics: {"server-negotiator.queue_time_us":206}
Aug 06 10:54:42 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:42 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.2@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:42.405807 8631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:42.395279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.2:47343)
0806 10:54:42.395973 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:42.395996 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:42.396169 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:42.396547 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:42.396551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:42.397021 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:42.397226 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:42.397236 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:42.398238 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:42.398243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:42.399185 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:42.399199 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:42.399951 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:42.399956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:42.400109 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:42.400739 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:42.400758 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:42.405438 (+ 4680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":282,"threads_started":1}
W20260806 10:54:42.406452 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:42.395422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:47343 (local address 127.30.218.62:39573)
0806 10:54:42.395669 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:42.395674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:42.396092 (+ 418us) server_negotiation.cc:407] Connection header received
0806 10:54:42.396312 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:42.396316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:42.396388 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:42.396475 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:42.397356 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:42.398096 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:42.399341 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:42.399796 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:42.399899 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:42.405977 (+ 6078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.2:47343: BlockingRecv error: recv got EOF from 127.30.218.2:47343 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 06 10:54:42 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 10:54:42 dist-test-slave-49h3 krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.218.1@KRBTEST.COM for oryx/127.30.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 10:54:42.765174 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:42.756158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39573 (local address 127.30.218.1:47459)
0806 10:54:42.756710 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:42.756730 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:42.756867 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:42.757201 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:42.757206 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:42.757691 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:42.757872 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:42.757882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:42.758930 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:42.758935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:42.759913 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:42.759930 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:42.760634 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:42.760641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:42.760808 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:42.761488 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:42.761513 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:42.764836 (+ 3323us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.218.62:39573: Server oryx/127.30.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":288,"threads_started":1}
W20260806 10:54:42.765359 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:42.756276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:47459 (local address 127.30.218.62:39573)
0806 10:54:42.756480 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:42.756486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:42.756810 (+ 324us) server_negotiation.cc:407] Connection header received
0806 10:54:42.756990 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:42.756996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:42.757061 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:42.757137 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:42.758014 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:42.758789 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:42.760046 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:42.760491 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:42.760602 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:42.765124 (+ 4522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.218.1:47459: BlockingRecv error: recv got EOF from 127.30.218.1:47459 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 10:54:42.829484 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 8165
I20260806 10:54:42.843861 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:42.830962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36174 (local address 127.30.218.1:44571)
0806 10:54:42.831760 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:42.831768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:42.831801 (+ 33us) server_negotiation.cc:407] Connection header received
0806 10:54:42.831869 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:42.831876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:42.832146 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:42.832326 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:42.833985 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:42.835138 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:42.836474 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:42.837108 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:42.837228 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:42.840111 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:42.840123 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:42.840152 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:42.840202 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:42.843226 (+ 3024us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.30.218.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 10:54:42.843595 (+ 369us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36174: Request ticket server kudu/127.30.218.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":442,"threads_started":1}
W20260806 10:54:42.843979 8634 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36174: Request ticket server kudu/127.30.218.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 10:54:42.844265 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 8165: Not authorized: Client connection negotiation failed: client connection to 127.30.218.1:44571: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.30.218.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 10:54:42.844460 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 8165
I20260806 10:54:42.862716 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 8318
I20260806 10:54:42.876920 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:42.863912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59510 (local address 127.30.218.2:35741)
0806 10:54:42.864714 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:42.864724 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:42.864760 (+ 36us) server_negotiation.cc:407] Connection header received
0806 10:54:42.864845 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:42.864855 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:42.865190 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:42.865477 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:42.866797 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:42.867964 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:42.869408 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:42.870099 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:42.870235 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:42.873016 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:42.873027 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:42.873045 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:42.873101 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:42.876109 (+ 3008us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.30.218.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 10:54:42.876547 (+ 438us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59510: Request ticket server kudu/127.30.218.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":398,"threads_started":1}
W20260806 10:54:42.877045 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 8318: Not authorized: Client connection negotiation failed: client connection to 127.30.218.2:35741: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.30.218.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 10:54:42.877072 8635 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59510: Request ticket server kudu/127.30.218.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 10:54:42.877234 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 8318
I20260806 10:54:42.894368 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 8453
I20260806 10:54:42.907693 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:42.895499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41916 (local address 127.30.218.3:39221)
0806 10:54:42.896212 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:42.896222 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:42.896252 (+ 30us) server_negotiation.cc:407] Connection header received
0806 10:54:42.896333 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:42.896342 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:42.896652 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:42.896867 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:42.898206 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:42.899348 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:42.901029 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:42.901667 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:42.901771 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:42.904350 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:42.904358 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:42.904373 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:42.904431 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:42.907103 (+ 2672us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.30.218.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 10:54:42.907433 (+ 330us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41916: Request ticket server kudu/127.30.218.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":337,"threads_started":1}
W20260806 10:54:42.907816 8636 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41916: Request ticket server kudu/127.30.218.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 10:54:42.907985 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 8453: Not authorized: Client connection negotiation failed: client connection to 127.30.218.3:39221: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.30.218.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 10:54:42.908150 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 8453
I20260806 10:54:42.925197 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 8096
I20260806 10:54:42.926653 8158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:43.056712 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 8096
2026-08-06T10:54:43Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21848 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T10:54:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:54:43Z Disabled control of system clock
I20260806 10:54:43.154690 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:46003
--webserver_interface=127.30.218.62
--webserver_port=0
--builtin_ntp_servers=127.30.218.20:34991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:46003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 10:54:43.499514 8641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:43.499848 8641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:43.499969 8641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:43.507558 8641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:54:43.507709 8641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:43.507778 8641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:43.507861 8641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:54:43.507922 8641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:54:43.517565 8641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:34991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:46003
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:46003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:43.518941 8641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:43.520800 8641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:43.532469 8647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:43.532428 8650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:43.532545 8648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:43.533124 8641 server_base.cc:1034] running on GCE node
I20260806 10:54:43.533973 8641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:43.535385 8641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:43.536588 8641 hybrid_clock.cc:648] HybridClock initialized: now 1786013683536548 us; error 31 us; skew 500 ppm
I20260806 10:54:43.536913 8641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:54:43.542734 8641 webserver.cc:460] Webserver started at http://127.30.218.62:40861/ using document root <none> and password file <none>
I20260806 10:54:43.543458 8641 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:43.543570 8641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:43.543908 8641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:43.546968 8641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "8e473f5411844e419e6dff0ddaf94b9c"
format_stamp: "Formatted at 2026-08-06 10:54:43 on dist-test-slave-49h3"
server_key: "a41e7371a2a7b6fdb5ab3f12b3ea3bec"
server_key_iv: "59bed9e2a75e228437e5e9cd318b9b87"
server_key_version: "encryptionkey@0"
I20260806 10:54:43.547809 8641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "8e473f5411844e419e6dff0ddaf94b9c"
format_stamp: "Formatted at 2026-08-06 10:54:43 on dist-test-slave-49h3"
server_key: "a41e7371a2a7b6fdb5ab3f12b3ea3bec"
server_key_iv: "59bed9e2a75e228437e5e9cd318b9b87"
server_key_version: "encryptionkey@0"
I20260806 10:54:43.553333 8641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 10:54:43.557121 8656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:43.558704 8641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:54:43.558893 8641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "8e473f5411844e419e6dff0ddaf94b9c"
format_stamp: "Formatted at 2026-08-06 10:54:43 on dist-test-slave-49h3"
server_key: "a41e7371a2a7b6fdb5ab3f12b3ea3bec"
server_key_iv: "59bed9e2a75e228437e5e9cd318b9b87"
server_key_version: "encryptionkey@0"
I20260806 10:54:43.559134 8641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:43.579221 8641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:43.580163 8641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:43.580468 8641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:43.598889 8641 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:46003
I20260806 10:54:43.598908 8708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:46003 every 8 connection(s)
I20260806 10:54:43.600251 8641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:54:43.603102 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 8641
I20260806 10:54:43.603333 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:54:43.604179 31592 external_mini_cluster.cc:1428] Setting key 8e34595b888d9cd79f81153899c011c6
I20260806 10:54:43.604614 8709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:43.618681 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:43.606291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46022 (local address 127.30.218.62:46003)
0806 10:54:43.607293 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:43.607303 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:43.607349 (+ 46us) server_negotiation.cc:407] Connection header received
0806 10:54:43.609353 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:43.609376 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:43.609922 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:43.610266 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:43.610967 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:43.612301 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:43.614071 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:43.614755 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:43.614901 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:43.616219 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:43.616230 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:43.616245 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:54:43.616854 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:43.616938 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:54:43.616964 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:43.617052 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:43.617293 (+ 241us) server_negotiation.cc:299] Negotiation successful
0806 10:54:43.617674 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":358,"threads_started":1}
I20260806 10:54:43.618738 8709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e473f5411844e419e6dff0ddaf94b9c: Bootstrap starting.
I20260806 10:54:43.622332 8709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e473f5411844e419e6dff0ddaf94b9c: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:43.623366 8709 log.cc:824] T 00000000000000000000000000000000 P 8e473f5411844e419e6dff0ddaf94b9c: Log is configured to *not* fsync() on all Append() calls
I20260806 10:54:43.626291 8709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e473f5411844e419e6dff0ddaf94b9c: No bootstrap required, opened a new log
I20260806 10:54:43.633644 8709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e473f5411844e419e6dff0ddaf94b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e473f5411844e419e6dff0ddaf94b9c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46003 } }
I20260806 10:54:43.634018 8709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e473f5411844e419e6dff0ddaf94b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:43.634124 8709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e473f5411844e419e6dff0ddaf94b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e473f5411844e419e6dff0ddaf94b9c, State: Initialized, Role: FOLLOWER
I20260806 10:54:43.634680 8709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e473f5411844e419e6dff0ddaf94b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e473f5411844e419e6dff0ddaf94b9c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46003 } }
I20260806 10:54:43.634889 8709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e473f5411844e419e6dff0ddaf94b9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:54:43.635035 8709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e473f5411844e419e6dff0ddaf94b9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:54:43.635216 8709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e473f5411844e419e6dff0ddaf94b9c [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:43.637430 8709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e473f5411844e419e6dff0ddaf94b9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e473f5411844e419e6dff0ddaf94b9c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46003 } }
I20260806 10:54:43.637965 8709 leader_election.cc:302] T 00000000000000000000000000000000 P 8e473f5411844e419e6dff0ddaf94b9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e473f5411844e419e6dff0ddaf94b9c; no voters:
I20260806 10:54:43.638509 8709 leader_election.cc:288] T 00000000000000000000000000000000 P 8e473f5411844e419e6dff0ddaf94b9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:54:43.638655 8714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e473f5411844e419e6dff0ddaf94b9c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:54:43.639322 8714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e473f5411844e419e6dff0ddaf94b9c [term 1 LEADER]: Becoming Leader. State: Replica: 8e473f5411844e419e6dff0ddaf94b9c, State: Running, Role: LEADER
I20260806 10:54:43.639842 8714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e473f5411844e419e6dff0ddaf94b9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e473f5411844e419e6dff0ddaf94b9c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46003 } }
I20260806 10:54:43.640401 8709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e473f5411844e419e6dff0ddaf94b9c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:54:43.642495 8716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e473f5411844e419e6dff0ddaf94b9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e473f5411844e419e6dff0ddaf94b9c. Latest consensus state: current_term: 1 leader_uuid: "8e473f5411844e419e6dff0ddaf94b9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e473f5411844e419e6dff0ddaf94b9c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46003 } } }
I20260806 10:54:43.642912 8716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e473f5411844e419e6dff0ddaf94b9c [sys.catalog]: This master's current role is: LEADER
I20260806 10:54:43.642834 8715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e473f5411844e419e6dff0ddaf94b9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e473f5411844e419e6dff0ddaf94b9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e473f5411844e419e6dff0ddaf94b9c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46003 } } }
I20260806 10:54:43.643136 8715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e473f5411844e419e6dff0ddaf94b9c [sys.catalog]: This master's current role is: LEADER
I20260806 10:54:43.645104 8723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:54:43.649340 8723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:54:43.658237 8723 catalog_manager.cc:1347] Generated new cluster ID: 997c48e08258454b97bac38b7c536438
I20260806 10:54:43.658387 8723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:54:43.693392 8723 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:54:43.694900 8723 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:54:43.702862 8723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e473f5411844e419e6dff0ddaf94b9c: Generated new TSK 0
I20260806 10:54:43.704016 8723 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:54:43.721277 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:46003
--builtin_ntp_servers=127.30.218.20:34991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 10:54:44.075378 8733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:44.075698 8733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:44.075824 8733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:44.083428 8733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:44.083539 8733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:44.083681 8733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:54:44.093158 8733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:34991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:46003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:44.094525 8733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:44.096279 8733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:44.110116 8739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:44.110044 8740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:44.110088 8742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:44.111338 8733 server_base.cc:1034] running on GCE node
I20260806 10:54:44.112044 8733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:44.113245 8733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:44.114475 8733 hybrid_clock.cc:648] HybridClock initialized: now 1786013684114411 us; error 65 us; skew 500 ppm
I20260806 10:54:44.114838 8733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:54:44.117308 8733 webserver.cc:460] Webserver started at http://127.30.218.1:36953/ using document root <none> and password file <none>
I20260806 10:54:44.118131 8733 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:44.118258 8733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:44.118594 8733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:44.121907 8733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "f807415a62b64b1e8f110bb8874cf8c9"
format_stamp: "Formatted at 2026-08-06 10:54:44 on dist-test-slave-49h3"
server_key: "66125e72fd4654dd1c76315e01b43b72"
server_key_iv: "550f209ace095750a04fee3620c5a584"
server_key_version: "encryptionkey@0"
I20260806 10:54:44.122833 8733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "f807415a62b64b1e8f110bb8874cf8c9"
format_stamp: "Formatted at 2026-08-06 10:54:44 on dist-test-slave-49h3"
server_key: "66125e72fd4654dd1c76315e01b43b72"
server_key_iv: "550f209ace095750a04fee3620c5a584"
server_key_version: "encryptionkey@0"
I20260806 10:54:44.128818 8733 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260806 10:54:44.132905 8748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:44.134493 8733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:54:44.134698 8733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "f807415a62b64b1e8f110bb8874cf8c9"
format_stamp: "Formatted at 2026-08-06 10:54:44 on dist-test-slave-49h3"
server_key: "66125e72fd4654dd1c76315e01b43b72"
server_key_iv: "550f209ace095750a04fee3620c5a584"
server_key_version: "encryptionkey@0"
I20260806 10:54:44.134946 8733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:44.173048 8733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:44.174041 8733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:44.174358 8733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:44.175386 8733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:54:44.176970 8733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:44.177088 8733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:44.177196 8733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:44.177265 8733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:44.209501 8733 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:34173
I20260806 10:54:44.209568 8861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:34173 every 8 connection(s)
I20260806 10:54:44.210855 8733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:54:44.217713 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 8733
I20260806 10:54:44.217950 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:54:44.218779 31592 external_mini_cluster.cc:1428] Setting key 4c387458d76c7ef7365c1b742b9e1158
I20260806 10:54:44.222041 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:46003
--builtin_ntp_servers=127.30.218.20:34991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 10:54:44.225082 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:44.213490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:44567 (local address 127.30.218.62:46003)
0806 10:54:44.213964 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:44.213969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:44.215323 (+ 1354us) server_negotiation.cc:407] Connection header received
0806 10:54:44.217242 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:44.217247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:44.217339 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:44.217439 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:44.219023 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:44.219901 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:44.222088 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:44.222597 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:44.222724 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:44.224337 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:44.224341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:44.224345 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:54:44.224364 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:44.224407 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:54:44.224420 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:44.224484 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:44.224730 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 10:54:44.224818 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":242,"threads_started":1}
I20260806 10:54:44.226528 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:44.214119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46003 (local address 127.30.218.1:44567)
0806 10:54:44.215091 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:44.215142 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:44.216924 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:44.217771 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:44.217785 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:44.217915 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:44.218822 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:44.218851 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:44.220054 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:44.220058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:44.221899 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:44.221917 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:44.222845 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:44.222853 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:44.223161 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:44.223939 (+ 778us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:54:44.223942 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:44.224001 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:54:44.224122 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:44.224502 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:44.224513 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:44.224520 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 10:54:44.224707 (+ 187us) client_negotiation.cc:240] Negotiation successful
0806 10:54:44.225144 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":284,"threads_started":1}
I20260806 10:54:44.228132 8862 heartbeater.cc:347] Connected to a master server at 127.30.218.62:46003
I20260806 10:54:44.228470 8862 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:44.229414 8862 heartbeater.cc:511] Master 127.30.218.62:46003 requested a full tablet report, sending...
I20260806 10:54:44.231704 8673 ts_manager.cc:194] Registered new tserver with Master: f807415a62b64b1e8f110bb8874cf8c9 (127.30.218.1:34173)
I20260806 10:54:44.233330 8673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.218.1:44567
W20260806 10:54:44.568040 8866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:44.568348 8866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:44.568459 8866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:44.575953 8866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:44.576068 8866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:44.576207 8866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:54:44.586404 8866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:34991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:46003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:44.587774 8866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:44.589660 8866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:44.602447 8872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:44.602982 8873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:44.603047 8875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:44.603492 8866 server_base.cc:1034] running on GCE node
I20260806 10:54:44.604261 8866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:44.605405 8866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:44.606689 8866 hybrid_clock.cc:648] HybridClock initialized: now 1786013684606622 us; error 51 us; skew 500 ppm
I20260806 10:54:44.607081 8866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:54:44.609705 8866 webserver.cc:460] Webserver started at http://127.30.218.2:42755/ using document root <none> and password file <none>
I20260806 10:54:44.610595 8866 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:44.610707 8866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:44.611052 8866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:44.614822 8866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "43ba3bd797be41409710fa974fccbf43"
format_stamp: "Formatted at 2026-08-06 10:54:44 on dist-test-slave-49h3"
server_key: "9291d2c92fd9604bbe934996b5733f33"
server_key_iv: "77937340b45f22699a08c68e7d655476"
server_key_version: "encryptionkey@0"
I20260806 10:54:44.615779 8866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "43ba3bd797be41409710fa974fccbf43"
format_stamp: "Formatted at 2026-08-06 10:54:44 on dist-test-slave-49h3"
server_key: "9291d2c92fd9604bbe934996b5733f33"
server_key_iv: "77937340b45f22699a08c68e7d655476"
server_key_version: "encryptionkey@0"
I20260806 10:54:44.621091 8866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260806 10:54:44.624230 8881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:44.625466 8866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 10:54:44.625622 8866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "43ba3bd797be41409710fa974fccbf43"
format_stamp: "Formatted at 2026-08-06 10:54:44 on dist-test-slave-49h3"
server_key: "9291d2c92fd9604bbe934996b5733f33"
server_key_iv: "77937340b45f22699a08c68e7d655476"
server_key_version: "encryptionkey@0"
I20260806 10:54:44.625813 8866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:44.660038 8866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:44.660952 8866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:44.661252 8866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:44.662263 8866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:54:44.663833 8866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:44.663935 8866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:44.664043 8866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:44.664111 8866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:44.693576 8866 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:40253
I20260806 10:54:44.693650 8994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:40253 every 8 connection(s)
I20260806 10:54:44.694952 8866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:54:44.704409 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 8866
I20260806 10:54:44.704599 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:54:44.705144 31592 external_mini_cluster.cc:1428] Setting key b8bbf8e305f34a6194b963bc9f591519
I20260806 10:54:44.707965 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:44.697432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:58921 (local address 127.30.218.62:46003)
0806 10:54:44.697684 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:44.697700 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:44.699103 (+ 1403us) server_negotiation.cc:407] Connection header received
0806 10:54:44.700861 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:44.700866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:44.700944 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:44.701033 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:44.702193 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:44.703101 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:44.704884 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:44.705366 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:44.705548 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:44.707119 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:44.707125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:44.707130 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:54:44.707147 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:44.707181 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:54:44.707194 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:44.707270 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:44.707535 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 10:54:44.707639 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 10:54:44.708041 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:46003
--builtin_ntp_servers=127.30.218.20:34991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 10:54:44.709203 8997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:44.698008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46003 (local address 127.30.218.2:58921)
0806 10:54:44.698922 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:44.698963 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:44.700613 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:44.701258 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:44.701266 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:44.701358 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:44.702030 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:44.702049 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:44.703292 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:44.703297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:44.704708 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:44.704724 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:44.705661 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:44.705669 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:44.705931 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:44.706745 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:54:44.706748 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:44.706794 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:54:44.706920 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:44.707284 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:44.707303 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:44.707312 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 10:54:44.707483 (+ 171us) client_negotiation.cc:240] Negotiation successful
0806 10:54:44.707970 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":280,"threads_started":1}
I20260806 10:54:44.710798 8995 heartbeater.cc:347] Connected to a master server at 127.30.218.62:46003
I20260806 10:54:44.711175 8995 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:44.712188 8995 heartbeater.cc:511] Master 127.30.218.62:46003 requested a full tablet report, sending...
I20260806 10:54:44.713955 8673 ts_manager.cc:194] Registered new tserver with Master: 43ba3bd797be41409710fa974fccbf43 (127.30.218.2:40253)
I20260806 10:54:44.714970 8673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.218.2:58921
W20260806 10:54:45.057833 8998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:45.058158 8998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:45.058252 8998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:45.066056 8998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:45.066195 8998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:45.066310 8998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:54:45.077742 8998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:34991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:46003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:45.079100 8998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:45.081039 8998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:45.094844 9005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:45.094870 9007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:45.094802 9004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:45.095933 8998 server_base.cc:1034] running on GCE node
I20260806 10:54:45.096616 8998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:45.097744 8998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:45.098964 8998 hybrid_clock.cc:648] HybridClock initialized: now 1786013685098896 us; error 57 us; skew 500 ppm
I20260806 10:54:45.099303 8998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:54:45.101630 8998 webserver.cc:460] Webserver started at http://127.30.218.3:35717/ using document root <none> and password file <none>
I20260806 10:54:45.102330 8998 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:45.102417 8998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:45.102699 8998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:45.105587 8998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "36e6165a00474d1cb8cf5a3c2ef78a65"
format_stamp: "Formatted at 2026-08-06 10:54:45 on dist-test-slave-49h3"
server_key: "b1ca3c8fa67747207fe2eb90bea9b475"
server_key_iv: "968ee50305e9f82d1d02be45c78c7b48"
server_key_version: "encryptionkey@0"
I20260806 10:54:45.106472 8998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "36e6165a00474d1cb8cf5a3c2ef78a65"
format_stamp: "Formatted at 2026-08-06 10:54:45 on dist-test-slave-49h3"
server_key: "b1ca3c8fa67747207fe2eb90bea9b475"
server_key_iv: "968ee50305e9f82d1d02be45c78c7b48"
server_key_version: "encryptionkey@0"
I20260806 10:54:45.111572 8998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:54:45.115306 9013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:45.116725 8998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 10:54:45.116881 8998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "36e6165a00474d1cb8cf5a3c2ef78a65"
format_stamp: "Formatted at 2026-08-06 10:54:45 on dist-test-slave-49h3"
server_key: "b1ca3c8fa67747207fe2eb90bea9b475"
server_key_iv: "968ee50305e9f82d1d02be45c78c7b48"
server_key_version: "encryptionkey@0"
I20260806 10:54:45.117105 8998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:45.133049 8998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:45.134040 8998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:45.134328 8998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:45.135290 8998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:54:45.136863 8998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:45.136955 8998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:45.137055 8998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:45.137111 8998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:45.169495 8998 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:40057
I20260806 10:54:45.169576 9127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:40057 every 8 connection(s)
I20260806 10:54:45.170872 8998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:54:45.171233 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 8998
I20260806 10:54:45.171406 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:54:45.171944 31592 external_mini_cluster.cc:1428] Setting key 9be016a58c5d6d0a55c8c1ba94839e5f
I20260806 10:54:45.183645 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:45.173433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:34023 (local address 127.30.218.62:46003)
0806 10:54:45.173682 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:45.173689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:45.175540 (+ 1851us) server_negotiation.cc:407] Connection header received
0806 10:54:45.177331 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:45.177336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:45.177413 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:45.177551 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:45.178614 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:45.179378 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:45.180964 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:45.181386 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:45.181539 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:45.182875 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:45.182880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:45.182883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:54:45.182906 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:45.182940 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:54:45.182952 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:45.183026 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:45.183285 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 10:54:45.183379 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 10:54:45.184818 9130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:45.174240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46003 (local address 127.30.218.3:34023)
0806 10:54:45.175330 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:45.175389 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:45.177043 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:45.177747 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:45.177756 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:45.177848 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:45.178456 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:45.178474 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:45.179513 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:45.179516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:45.180810 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:45.180823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:45.181631 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:45.181637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:45.181866 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:45.182536 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:54:45.182539 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:45.182591 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:54:45.182679 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:45.183076 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:45.183087 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:45.183094 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 10:54:45.183267 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 10:54:45.183642 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":337,"threads_started":1}
I20260806 10:54:45.186372 9128 heartbeater.cc:347] Connected to a master server at 127.30.218.62:46003
I20260806 10:54:45.186730 9128 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:45.187716 9128 heartbeater.cc:511] Master 127.30.218.62:46003 requested a full tablet report, sending...
I20260806 10:54:45.189188 8673 ts_manager.cc:194] Registered new tserver with Master: 36e6165a00474d1cb8cf5a3c2ef78a65 (127.30.218.3:40057)
I20260806 10:54:45.190035 8673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.218.3:34023
I20260806 10:54:45.200191 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 10:54:45.206317 8864 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:45.204620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46038 (local address 127.30.218.62:46003)
0806 10:54:45.204829 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:45.204836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:45.205224 (+ 388us) server_negotiation.cc:407] Connection header received
0806 10:54:45.205389 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:45.205393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:45.205503 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:45.205585 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:45.206057 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46038: BlockingRecv error: recv got EOF from 127.0.0.1:46038 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 10:54:45.206768 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 8733
I20260806 10:54:45.215704 9137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:45.207701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.30.218.1:34173)
0806 10:54:45.208171 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:45.208181 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:45.208211 (+ 30us) server_negotiation.cc:407] Connection header received
0806 10:54:45.208274 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:45.208281 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:45.208484 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:45.208625 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:45.208956 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:45.210599 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:45.211948 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:45.212808 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:45.212947 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:45.214074 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:45.214085 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:45.214102 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:54:45.214837 (+ 735us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:45.214897 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:54:45.214922 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:45.215038 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:45.215184 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 10:54:45.215320 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":232,"threads_started":1}
I20260806 10:54:45.216861 8857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:45.298782 8862 heartbeater.cc:503] Master 127.30.218.62:46003 was elected leader, sending a full tablet report...
I20260806 10:54:45.299865 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 8733
W20260806 10:54:45.315778 8657 connection.cc:535] server connection from 127.30.218.1:44567 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 10:54:45.316161 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 8866
I20260806 10:54:45.323849 9139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:45.317084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42204 (local address 127.30.218.2:40253)
0806 10:54:45.317609 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:45.317617 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:45.317645 (+ 28us) server_negotiation.cc:407] Connection header received
0806 10:54:45.317720 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:45.317727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:45.317947 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:45.318082 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:45.318415 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:45.319390 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:45.320638 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:45.321252 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:45.321359 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:45.322430 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:45.322439 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:45.322452 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:54:45.323057 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:45.323112 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:54:45.323147 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:45.323237 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:45.323339 (+ 102us) server_negotiation.cc:299] Negotiation successful
0806 10:54:45.323464 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":248,"threads_started":1}
I20260806 10:54:45.325176 8990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:45.408802 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 8866
I20260806 10:54:45.426120 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 8998
I20260806 10:54:45.434227 9141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:45.427157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.30.218.3:40057)
0806 10:54:45.427834 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:45.427841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:45.427873 (+ 32us) server_negotiation.cc:407] Connection header received
0806 10:54:45.427942 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:45.427949 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:45.428193 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:45.428332 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:45.428743 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:45.429872 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:45.431056 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:45.431617 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:45.431707 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:45.432709 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:45.432718 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:45.432732 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:54:45.433328 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:45.433383 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:54:45.433403 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:45.433559 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:45.433698 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 10:54:45.433830 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":367,"threads_started":1}
I20260806 10:54:45.435482 9122 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:45.519954 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 8998
I20260806 10:54:45.536214 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 8641
I20260806 10:54:45.537518 8704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:45.645771 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 8641
2026-08-06T10:54:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2580 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T10:54:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:54:45Z Disabled control of system clock
I20260806 10:54:45.732360 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:45217
--webserver_interface=127.30.218.62
--webserver_port=0
--builtin_ntp_servers=127.30.218.20:43933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:45217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 10:54:46.090469 9147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:46.090786 9147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:46.090914 9147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:46.098690 9147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:54:46.098830 9147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:46.098893 9147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:46.098978 9147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:54:46.099035 9147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:54:46.108811 9147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:43933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:45217
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:45217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:46.110183 9147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:46.111999 9147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:46.123263 9152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:46.123343 9153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:46.123672 9155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:46.123994 9147 server_base.cc:1034] running on GCE node
I20260806 10:54:46.124678 9147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:46.125996 9147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:46.127214 9147 hybrid_clock.cc:648] HybridClock initialized: now 1786013686127168 us; error 37 us; skew 500 ppm
I20260806 10:54:46.127527 9147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:54:46.129662 9147 webserver.cc:460] Webserver started at http://127.30.218.62:35375/ using document root <none> and password file <none>
I20260806 10:54:46.130302 9147 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:46.130405 9147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:46.130710 9147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:46.133697 9147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "5724ece2f1544078ac699b20110fa79d"
format_stamp: "Formatted at 2026-08-06 10:54:46 on dist-test-slave-49h3"
server_key: "09eab103c22e7c5125763d6b36aa80a1"
server_key_iv: "8f40cc41169472e61739d2f7b0917107"
server_key_version: "encryptionkey@0"
I20260806 10:54:46.134465 9147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "5724ece2f1544078ac699b20110fa79d"
format_stamp: "Formatted at 2026-08-06 10:54:46 on dist-test-slave-49h3"
server_key: "09eab103c22e7c5125763d6b36aa80a1"
server_key_iv: "8f40cc41169472e61739d2f7b0917107"
server_key_version: "encryptionkey@0"
I20260806 10:54:46.139931 9147 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260806 10:54:46.143621 9161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:46.145090 9147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:54:46.145283 9147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "5724ece2f1544078ac699b20110fa79d"
format_stamp: "Formatted at 2026-08-06 10:54:46 on dist-test-slave-49h3"
server_key: "09eab103c22e7c5125763d6b36aa80a1"
server_key_iv: "8f40cc41169472e61739d2f7b0917107"
server_key_version: "encryptionkey@0"
I20260806 10:54:46.145536 9147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:46.151865 9147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:46.152683 9147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:46.152974 9147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:46.170307 9147 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:45217
I20260806 10:54:46.170348 9213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:45217 every 8 connection(s)
I20260806 10:54:46.171617 9147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:54:46.173749 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 9147
I20260806 10:54:46.173957 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:54:46.174790 31592 external_mini_cluster.cc:1428] Setting key 23c09b29e804567b0f5c17411c80aa8b
I20260806 10:54:46.175974 9214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:46.183168 9218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:46.176933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45302 (local address 127.30.218.62:45217)
0806 10:54:46.177928 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:46.177939 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:46.177993 (+ 54us) server_negotiation.cc:407] Connection header received
0806 10:54:46.179480 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:46.179501 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:46.179911 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:46.180245 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:46.181391 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:46.181398 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:46.181411 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:54:46.181429 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:46.181567 (+ 138us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:54:46.181587 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:46.181792 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:46.181899 (+ 107us) server_negotiation.cc:299] Negotiation successful
0806 10:54:46.182177 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":349,"threads_started":1}
I20260806 10:54:46.189656 9214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5724ece2f1544078ac699b20110fa79d: Bootstrap starting.
I20260806 10:54:46.192924 9214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5724ece2f1544078ac699b20110fa79d: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:46.194118 9214 log.cc:824] T 00000000000000000000000000000000 P 5724ece2f1544078ac699b20110fa79d: Log is configured to *not* fsync() on all Append() calls
I20260806 10:54:46.197429 9214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5724ece2f1544078ac699b20110fa79d: No bootstrap required, opened a new log
I20260806 10:54:46.205011 9214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5724ece2f1544078ac699b20110fa79d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5724ece2f1544078ac699b20110fa79d" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 45217 } }
I20260806 10:54:46.205479 9214 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5724ece2f1544078ac699b20110fa79d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:46.205586 9214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5724ece2f1544078ac699b20110fa79d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5724ece2f1544078ac699b20110fa79d, State: Initialized, Role: FOLLOWER
I20260806 10:54:46.206235 9214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5724ece2f1544078ac699b20110fa79d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5724ece2f1544078ac699b20110fa79d" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 45217 } }
I20260806 10:54:46.206481 9214 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5724ece2f1544078ac699b20110fa79d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:54:46.206617 9214 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5724ece2f1544078ac699b20110fa79d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:54:46.206782 9214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5724ece2f1544078ac699b20110fa79d [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:46.208985 9214 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5724ece2f1544078ac699b20110fa79d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5724ece2f1544078ac699b20110fa79d" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 45217 } }
I20260806 10:54:46.209609 9214 leader_election.cc:302] T 00000000000000000000000000000000 P 5724ece2f1544078ac699b20110fa79d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5724ece2f1544078ac699b20110fa79d; no voters:
I20260806 10:54:46.210282 9214 leader_election.cc:288] T 00000000000000000000000000000000 P 5724ece2f1544078ac699b20110fa79d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:54:46.210407 9220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5724ece2f1544078ac699b20110fa79d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:54:46.211103 9220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5724ece2f1544078ac699b20110fa79d [term 1 LEADER]: Becoming Leader. State: Replica: 5724ece2f1544078ac699b20110fa79d, State: Running, Role: LEADER
I20260806 10:54:46.211650 9220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5724ece2f1544078ac699b20110fa79d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5724ece2f1544078ac699b20110fa79d" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 45217 } }
I20260806 10:54:46.212352 9214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5724ece2f1544078ac699b20110fa79d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:54:46.214551 9222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5724ece2f1544078ac699b20110fa79d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5724ece2f1544078ac699b20110fa79d. Latest consensus state: current_term: 1 leader_uuid: "5724ece2f1544078ac699b20110fa79d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5724ece2f1544078ac699b20110fa79d" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 45217 } } }
I20260806 10:54:46.214634 9221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5724ece2f1544078ac699b20110fa79d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5724ece2f1544078ac699b20110fa79d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5724ece2f1544078ac699b20110fa79d" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 45217 } } }
I20260806 10:54:46.214946 9222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5724ece2f1544078ac699b20110fa79d [sys.catalog]: This master's current role is: LEADER
I20260806 10:54:46.215010 9221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5724ece2f1544078ac699b20110fa79d [sys.catalog]: This master's current role is: LEADER
I20260806 10:54:46.216609 9226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:54:46.221997 9226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:54:46.230790 9226 catalog_manager.cc:1347] Generated new cluster ID: 57ac548b153d47278ef9d11c8448c42c
I20260806 10:54:46.230945 9226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:54:46.243062 9226 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:54:46.243520 9226 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:54:46.252108 9226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5724ece2f1544078ac699b20110fa79d: Generated new TSK 0
I20260806 10:54:46.252974 9226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:54:46.266263 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:45217
--builtin_ntp_servers=127.30.218.20:43933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 10:54:46.603335 9239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:46.603657 9239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:46.603777 9239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:46.611402 9239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:46.611537 9239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:46.611685 9239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:54:46.621304 9239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:43933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:45217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:46.622648 9239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:46.624528 9239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:46.638167 9245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:46.638216 9247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:46.638247 9244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:46.639076 9239 server_base.cc:1034] running on GCE node
I20260806 10:54:46.639686 9239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:46.640731 9239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:46.641971 9239 hybrid_clock.cc:648] HybridClock initialized: now 1786013686641904 us; error 64 us; skew 500 ppm
I20260806 10:54:46.642387 9239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:54:46.644966 9239 webserver.cc:460] Webserver started at http://127.30.218.1:35735/ using document root <none> and password file <none>
I20260806 10:54:46.645831 9239 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:46.645951 9239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:46.646313 9239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:46.649647 9239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "545566411e4940bab47b4f368ea83340"
format_stamp: "Formatted at 2026-08-06 10:54:46 on dist-test-slave-49h3"
server_key: "77c2a3e79a84076fb5a5625b25130bd5"
server_key_iv: "2082a309b07d60062b838017a544023e"
server_key_version: "encryptionkey@0"
I20260806 10:54:46.650646 9239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "545566411e4940bab47b4f368ea83340"
format_stamp: "Formatted at 2026-08-06 10:54:46 on dist-test-slave-49h3"
server_key: "77c2a3e79a84076fb5a5625b25130bd5"
server_key_iv: "2082a309b07d60062b838017a544023e"
server_key_version: "encryptionkey@0"
I20260806 10:54:46.656705 9239 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260806 10:54:46.660795 9253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:46.662330 9239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:54:46.662506 9239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "545566411e4940bab47b4f368ea83340"
format_stamp: "Formatted at 2026-08-06 10:54:46 on dist-test-slave-49h3"
server_key: "77c2a3e79a84076fb5a5625b25130bd5"
server_key_iv: "2082a309b07d60062b838017a544023e"
server_key_version: "encryptionkey@0"
I20260806 10:54:46.662750 9239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:46.668926 9239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:46.669859 9239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:46.670142 9239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:46.671173 9239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:54:46.672721 9239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:46.672822 9239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:46.672938 9239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:46.673010 9239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:46.705925 9239 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:46471
I20260806 10:54:46.705989 9366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:46471 every 8 connection(s)
I20260806 10:54:46.707345 9239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:54:46.716017 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:46.710012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:43411 (local address 127.30.218.62:45217)
0806 10:54:46.710498 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:46.710504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:46.711780 (+ 1276us) server_negotiation.cc:407] Connection header received
0806 10:54:46.713610 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:46.713615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:46.713674 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:46.713781 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:46.715120 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:46.715125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:46.715128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:54:46.715147 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:46.715220 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:54:46.715233 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:46.715396 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:46.715621 (+ 225us) server_negotiation.cc:299] Negotiation successful
0806 10:54:46.715695 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":230,"threads_started":1}
I20260806 10:54:46.716997 9370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:46.710613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:45217 (local address 127.30.218.1:43411)
0806 10:54:46.711583 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:46.711628 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:46.713254 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:46.713999 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:46.714008 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:46.714105 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:46.714793 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:54:46.714796 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:46.714837 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:54:46.714928 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:46.715371 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:46.715384 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:46.715392 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 10:54:46.715695 (+ 303us) client_negotiation.cc:240] Negotiation successful
0806 10:54:46.715983 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":310,"threads_started":1}
I20260806 10:54:46.717998 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 9239
I20260806 10:54:46.718230 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:54:46.718708 9367 heartbeater.cc:347] Connected to a master server at 127.30.218.62:45217
I20260806 10:54:46.718918 31592 external_mini_cluster.cc:1428] Setting key 5de889cdb0ae2d459f8f48710f3921ff
I20260806 10:54:46.719022 9367 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:46.719645 9367 heartbeater.cc:511] Master 127.30.218.62:45217 requested a full tablet report, sending...
I20260806 10:54:46.721730 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:45217
--builtin_ntp_servers=127.30.218.20:43933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 10:54:46.721707 9178 ts_manager.cc:194] Registered new tserver with Master: 545566411e4940bab47b4f368ea83340 (127.30.218.1:46471)
W20260806 10:54:47.066546 9371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:47.066855 9371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:47.067001 9371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:47.074417 9371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:47.074538 9371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:47.074698 9371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:54:47.085484 9371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:43933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:45217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:47.086810 9371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:47.088603 9371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:47.101435 9376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:47.101521 9377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:47.101995 9379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:47.102774 9371 server_base.cc:1034] running on GCE node
I20260806 10:54:47.103442 9371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:47.104456 9371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:47.105656 9371 hybrid_clock.cc:648] HybridClock initialized: now 1786013687105605 us; error 41 us; skew 500 ppm
I20260806 10:54:47.105988 9371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:54:47.108371 9371 webserver.cc:460] Webserver started at http://127.30.218.2:43995/ using document root <none> and password file <none>
I20260806 10:54:47.109104 9371 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:47.109221 9371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:47.109586 9371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:47.112780 9371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "e1239aadf4d64a9fb84d5be07d03e73d"
format_stamp: "Formatted at 2026-08-06 10:54:47 on dist-test-slave-49h3"
server_key: "abbe832515e9a647c7165f468053577d"
server_key_iv: "4678a60a12ba4dcc38f81c2e191cfeb5"
server_key_version: "encryptionkey@0"
I20260806 10:54:47.113807 9371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "e1239aadf4d64a9fb84d5be07d03e73d"
format_stamp: "Formatted at 2026-08-06 10:54:47 on dist-test-slave-49h3"
server_key: "abbe832515e9a647c7165f468053577d"
server_key_iv: "4678a60a12ba4dcc38f81c2e191cfeb5"
server_key_version: "encryptionkey@0"
I20260806 10:54:47.119400 9371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:54:47.123219 9385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:47.124600 9371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 10:54:47.124769 9371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "e1239aadf4d64a9fb84d5be07d03e73d"
format_stamp: "Formatted at 2026-08-06 10:54:47 on dist-test-slave-49h3"
server_key: "abbe832515e9a647c7165f468053577d"
server_key_iv: "4678a60a12ba4dcc38f81c2e191cfeb5"
server_key_version: "encryptionkey@0"
I20260806 10:54:47.125006 9371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:47.131058 9371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:47.132014 9371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:47.132306 9371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:47.133314 9371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:54:47.134893 9371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:47.134996 9371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:47.135119 9371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:47.135188 9371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:47.165390 9371 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:40473
I20260806 10:54:47.165468 9498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:40473 every 8 connection(s)
I20260806 10:54:47.166864 9371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:54:47.172730 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 9371
I20260806 10:54:47.172916 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:54:47.173502 31592 external_mini_cluster.cc:1428] Setting key 8194a90f3fc38c6ded3c756caa797d57
I20260806 10:54:47.176599 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:47.169664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:40073 (local address 127.30.218.62:45217)
0806 10:54:47.169909 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:47.169915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:47.171466 (+ 1551us) server_negotiation.cc:407] Connection header received
0806 10:54:47.173811 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:47.173817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:47.173888 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:47.174012 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:47.175794 (+ 1782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:47.175800 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:47.175806 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:54:47.175828 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:47.175872 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:54:47.175888 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:47.176005 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:47.176225 (+ 220us) server_negotiation.cc:299] Negotiation successful
0806 10:54:47.176288 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260806 10:54:47.176695 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:45217
--builtin_ntp_servers=127.30.218.20:43933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 10:54:47.177946 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:47.170160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:45217 (local address 127.30.218.2:40073)
0806 10:54:47.171274 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:47.171328 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:47.173425 (+ 2097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:47.174490 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:47.174500 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:47.174619 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:47.175441 (+ 822us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:54:47.175443 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:47.175486 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:54:47.175608 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:47.176000 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:47.176011 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:47.176019 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 10:54:47.176321 (+ 302us) client_negotiation.cc:240] Negotiation successful
0806 10:54:47.176644 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":337,"threads_started":1}
I20260806 10:54:47.185631 9499 heartbeater.cc:347] Connected to a master server at 127.30.218.62:45217
I20260806 10:54:47.186216 9499 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:47.187060 9499 heartbeater.cc:511] Master 127.30.218.62:45217 requested a full tablet report, sending...
I20260806 10:54:47.189112 9178 ts_manager.cc:194] Registered new tserver with Master: e1239aadf4d64a9fb84d5be07d03e73d (127.30.218.2:40473)
W20260806 10:54:47.533099 9502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:47.533418 9502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:47.533556 9502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:47.540952 9502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:47.541055 9502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:47.541182 9502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:54:47.550354 9502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:43933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:45217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:47.551839 9502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:47.553941 9502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:47.568606 9510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:47.568610 9507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:47.568892 9508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:47.569619 9502 server_base.cc:1034] running on GCE node
I20260806 10:54:47.570331 9502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:47.571379 9502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:47.572579 9502 hybrid_clock.cc:648] HybridClock initialized: now 1786013687572526 us; error 45 us; skew 500 ppm
I20260806 10:54:47.572930 9502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:54:47.575455 9502 webserver.cc:460] Webserver started at http://127.30.218.3:33143/ using document root <none> and password file <none>
I20260806 10:54:47.576215 9502 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:47.576318 9502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:47.576723 9502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:47.579981 9502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "8b3093866bce44f2a875dc52ff9d31ff"
format_stamp: "Formatted at 2026-08-06 10:54:47 on dist-test-slave-49h3"
server_key: "bd3e9f2878af8faca48508dd3144b357"
server_key_iv: "06ce767d8ca5de8969a88090dac3d4ae"
server_key_version: "encryptionkey@0"
I20260806 10:54:47.580833 9502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "8b3093866bce44f2a875dc52ff9d31ff"
format_stamp: "Formatted at 2026-08-06 10:54:47 on dist-test-slave-49h3"
server_key: "bd3e9f2878af8faca48508dd3144b357"
server_key_iv: "06ce767d8ca5de8969a88090dac3d4ae"
server_key_version: "encryptionkey@0"
I20260806 10:54:47.586243 9502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260806 10:54:47.589965 9516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:47.591463 9502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:54:47.591624 9502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "8b3093866bce44f2a875dc52ff9d31ff"
format_stamp: "Formatted at 2026-08-06 10:54:47 on dist-test-slave-49h3"
server_key: "bd3e9f2878af8faca48508dd3144b357"
server_key_iv: "06ce767d8ca5de8969a88090dac3d4ae"
server_key_version: "encryptionkey@0"
I20260806 10:54:47.591827 9502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:47.597770 9502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:47.598588 9502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:47.598840 9502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:47.599773 9502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:54:47.601290 9502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:47.601387 9502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:47.601495 9502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:47.601560 9502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:47.635506 9502 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:34755
I20260806 10:54:47.635547 9631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:34755 every 8 connection(s)
I20260806 10:54:47.637043 9502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:54:47.638562 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 9502
I20260806 10:54:47.638742 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:54:47.639267 31592 external_mini_cluster.cc:1428] Setting key 9714b5025285a5868eaf22f71b6e997d
I20260806 10:54:47.645800 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:47.639941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:57559 (local address 127.30.218.62:45217)
0806 10:54:47.640193 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:47.640200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:47.641897 (+ 1697us) server_negotiation.cc:407] Connection header received
0806 10:54:47.643712 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:47.643717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:47.643779 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:47.643869 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:47.645129 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:47.645134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:47.645137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:54:47.645153 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:47.645190 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:54:47.645203 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:47.645308 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:47.645484 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 10:54:47.645548 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 10:54:47.646739 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:47.640527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:45217 (local address 127.30.218.3:57559)
0806 10:54:47.641720 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:47.641769 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:47.643459 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:47.644069 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:47.644077 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:47.644170 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:47.644852 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:54:47.644854 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:47.644895 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:54:47.644981 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:47.645274 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:47.645280 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:47.645285 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 10:54:47.645548 (+ 263us) client_negotiation.cc:240] Negotiation successful
0806 10:54:47.645758 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":354,"threads_started":1}
I20260806 10:54:47.648085 9632 heartbeater.cc:347] Connected to a master server at 127.30.218.62:45217
I20260806 10:54:47.648344 9632 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:47.648932 9632 heartbeater.cc:511] Master 127.30.218.62:45217 requested a full tablet report, sending...
I20260806 10:54:47.650550 9178 ts_manager.cc:194] Registered new tserver with Master: 8b3093866bce44f2a875dc52ff9d31ff (127.30.218.3:34755)
I20260806 10:54:47.654687 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 10:54:47.661371 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:54:47.659654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45306 (local address 127.30.218.62:45217)
0806 10:54:47.659823 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:47.659830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:47.660324 (+ 494us) server_negotiation.cc:407] Connection header received
0806 10:54:47.660496 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:47.660503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:47.660569 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:47.660656 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:47.661086 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45306: BlockingRecv error: recv got EOF from 127.0.0.1:45306 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 10:54:47.662173 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 9239
I20260806 10:54:47.666074 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:47.663245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53116 (local address 127.30.218.1:46471)
0806 10:54:47.663857 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:47.663866 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:47.663896 (+ 30us) server_negotiation.cc:407] Connection header received
0806 10:54:47.663976 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:47.663982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:47.664234 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:47.664373 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:47.665370 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:47.665377 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:47.665388 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:54:47.665407 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:47.665480 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:54:47.665502 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:47.665624 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:47.665691 (+ 67us) server_negotiation.cc:299] Negotiation successful
0806 10:54:47.665789 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":290,"threads_started":1}
I20260806 10:54:47.667472 9362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:47.752614 9367 heartbeater.cc:503] Master 127.30.218.62:45217 was elected leader, sending a full tablet report...
I20260806 10:54:47.754269 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 9239
W20260806 10:54:47.774647 9164 connection.cc:535] server connection from 127.30.218.1:43411 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 10:54:47.774897 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 9371
I20260806 10:54:47.779325 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:47.776075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40800 (local address 127.30.218.2:40473)
0806 10:54:47.776658 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:47.776667 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:47.776703 (+ 36us) server_negotiation.cc:407] Connection header received
0806 10:54:47.776783 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:47.776791 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:47.777094 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:47.777250 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:47.778515 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:47.778524 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:47.778538 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:54:47.778554 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:47.778613 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:54:47.778635 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:47.778762 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:47.778863 (+ 101us) server_negotiation.cc:299] Negotiation successful
0806 10:54:47.778993 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":266,"threads_started":1}
I20260806 10:54:47.780910 9494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:47.867434 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 9371
I20260806 10:54:47.886108 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 9502
I20260806 10:54:47.890903 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:47.887437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48968 (local address 127.30.218.3:34755)
0806 10:54:47.888280 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:47.888298 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:47.888366 (+ 68us) server_negotiation.cc:407] Connection header received
0806 10:54:47.888447 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:47.888455 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:47.888743 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:47.888891 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:47.890024 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:47.890032 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:47.890045 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:54:47.890057 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:47.890109 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:54:47.890125 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:47.890271 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:47.890413 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 10:54:47.890532 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":448,"threads_started":1}
I20260806 10:54:47.892354 9627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:47.984748 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 9502
I20260806 10:54:48.004123 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 9147
I20260806 10:54:48.005610 9209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:48.125370 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 9147
2026-08-06T10:54:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2483 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:54:48 dist-test-slave-49h3 krb5kdc[9650](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:54:48 dist-test-slave-49h3 krb5kdc[9650](info): set up 2 sockets
Aug 06 10:54:48 dist-test-slave-49h3 krb5kdc[9650](info): commencing operation
krb5kdc: starting...
W20260806 10:54:49.211745 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.002s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:54:49 dist-test-slave-49h3 krb5kdc[9650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:54:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:54:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:54:49.524439 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:39961
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:34637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:39961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:54:49.880878 9664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:49.881232 9664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:49.881381 9664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:49.888705 9664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:54:49.888826 9664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:49.888908 9664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:49.888984 9664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:54:49.889045 9664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:54:49.899418 9664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:34637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:39961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:39961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:49.900946 9664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:49.902395 9664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:49.914320 9672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:49.914405 9670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:49.914405 9669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:49.915747 9664 server_base.cc:1034] running on GCE node
I20260806 10:54:49.916512 9664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:49.918047 9664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:49.919301 9664 hybrid_clock.cc:648] HybridClock initialized: now 1786013689919240 us; error 41 us; skew 500 ppm
I20260806 10:54:49.919638 9664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:49 dist-test-slave-49h3 krb5kdc[9650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013689, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:49.925702 9664 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:54:49.927234 9664 webserver.cc:460] Webserver started at http://127.30.218.62:40015/ using document root <none> and password file <none>
I20260806 10:54:49.927937 9664 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:49.928048 9664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:49.928318 9664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:49.931581 9664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "89172c9a7ac6402cacb26270870be367"
format_stamp: "Formatted at 2026-08-06 10:54:49 on dist-test-slave-49h3"
server_key: "7cf72d00e3a9c0a72ba57a8c07f3408e"
server_key_iv: "6644a17dae3ae6b44478a0bf2217b3df"
server_key_version: "encryptionkey@0"
I20260806 10:54:49.932355 9664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "89172c9a7ac6402cacb26270870be367"
format_stamp: "Formatted at 2026-08-06 10:54:49 on dist-test-slave-49h3"
server_key: "7cf72d00e3a9c0a72ba57a8c07f3408e"
server_key_iv: "6644a17dae3ae6b44478a0bf2217b3df"
server_key_version: "encryptionkey@0"
I20260806 10:54:49.939081 9664 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260806 10:54:49.943859 9679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:49.945258 9664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:54:49.945436 9664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "89172c9a7ac6402cacb26270870be367"
format_stamp: "Formatted at 2026-08-06 10:54:49 on dist-test-slave-49h3"
server_key: "7cf72d00e3a9c0a72ba57a8c07f3408e"
server_key_iv: "6644a17dae3ae6b44478a0bf2217b3df"
server_key_version: "encryptionkey@0"
I20260806 10:54:49.945693 9664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:49.958650 9664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:49.963274 9664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:49.963585 9664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:49.982509 9664 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:39961
I20260806 10:54:49.982540 9730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:39961 every 8 connection(s)
I20260806 10:54:49.984103 9664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:54:49.986001 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 9664
I20260806 10:54:49.986223 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:54:49.986923 31592 external_mini_cluster.cc:1428] Setting key 56dd072ac983ea8d018f50a62dd96aa4
I20260806 10:54:49.988376 9731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 10:54:49 dist-test-slave-49h3 krb5kdc[9650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:54:50.003396 9731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89172c9a7ac6402cacb26270870be367: Bootstrap starting.
I20260806 10:54:50.007175 9731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89172c9a7ac6402cacb26270870be367: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:50.008558 9731 log.cc:824] T 00000000000000000000000000000000 P 89172c9a7ac6402cacb26270870be367: Log is configured to *not* fsync() on all Append() calls
I20260806 10:54:50.011873 9734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:49.989058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43992 (local address 127.30.218.62:39961)
0806 10:54:49.989910 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:49.989923 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:49.989966 (+ 43us) server_negotiation.cc:407] Connection header received
0806 10:54:49.991636 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:49.991660 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:49.992109 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:49.992457 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:49.994605 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:49.996029 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:49.997833 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:49.998578 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:49.998746 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:50.003896 (+ 5150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:50.003907 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:50.003925 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:50.003973 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:50.007253 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:50.008270 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:50.008278 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:50.008285 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:50.008481 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:50.008935 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:50.008939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:50.008941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:50.009748 (+ 807us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:50.010143 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:50.010413 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 10:54:50.010837 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":400,"threads_started":1}
I20260806 10:54:50.012063 9731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89172c9a7ac6402cacb26270870be367: No bootstrap required, opened a new log
I20260806 10:54:50.019686 9731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89172c9a7ac6402cacb26270870be367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89172c9a7ac6402cacb26270870be367" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 39961 } }
I20260806 10:54:50.020115 9731 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89172c9a7ac6402cacb26270870be367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:50.020226 9731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89172c9a7ac6402cacb26270870be367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89172c9a7ac6402cacb26270870be367, State: Initialized, Role: FOLLOWER
I20260806 10:54:50.020849 9731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89172c9a7ac6402cacb26270870be367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89172c9a7ac6402cacb26270870be367" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 39961 } }
I20260806 10:54:50.021090 9731 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89172c9a7ac6402cacb26270870be367 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:54:50.021256 9731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89172c9a7ac6402cacb26270870be367 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:54:50.021428 9731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89172c9a7ac6402cacb26270870be367 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:50.023993 9731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89172c9a7ac6402cacb26270870be367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89172c9a7ac6402cacb26270870be367" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 39961 } }
I20260806 10:54:50.024611 9731 leader_election.cc:302] T 00000000000000000000000000000000 P 89172c9a7ac6402cacb26270870be367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89172c9a7ac6402cacb26270870be367; no voters:
I20260806 10:54:50.025274 9731 leader_election.cc:288] T 00000000000000000000000000000000 P 89172c9a7ac6402cacb26270870be367 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:54:50.025424 9736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89172c9a7ac6402cacb26270870be367 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:54:50.026192 9736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89172c9a7ac6402cacb26270870be367 [term 1 LEADER]: Becoming Leader. State: Replica: 89172c9a7ac6402cacb26270870be367, State: Running, Role: LEADER
I20260806 10:54:50.026728 9736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89172c9a7ac6402cacb26270870be367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89172c9a7ac6402cacb26270870be367" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 39961 } }
I20260806 10:54:50.027480 9731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89172c9a7ac6402cacb26270870be367 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:54:50.029542 9738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89172c9a7ac6402cacb26270870be367 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89172c9a7ac6402cacb26270870be367. Latest consensus state: current_term: 1 leader_uuid: "89172c9a7ac6402cacb26270870be367" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89172c9a7ac6402cacb26270870be367" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 39961 } } }
I20260806 10:54:50.029492 9737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89172c9a7ac6402cacb26270870be367 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89172c9a7ac6402cacb26270870be367" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89172c9a7ac6402cacb26270870be367" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 39961 } } }
I20260806 10:54:50.030072 9738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89172c9a7ac6402cacb26270870be367 [sys.catalog]: This master's current role is: LEADER
I20260806 10:54:50.030095 9737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89172c9a7ac6402cacb26270870be367 [sys.catalog]: This master's current role is: LEADER
I20260806 10:54:50.031023 9741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:54:50.036826 9741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:54:50.047168 9741 catalog_manager.cc:1347] Generated new cluster ID: a53b4ba7abcb4322a9bd3e891832cebb
I20260806 10:54:50.047317 9741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:54:50.058252 9741 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:54:50.060309 9741 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:54:50.070498 9741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89172c9a7ac6402cacb26270870be367: Generated new TSK 0
I20260806 10:54:50.071635 9741 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
I20260806 10:54:50.153625 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:39961
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:34637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:54:50.497572 9757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:50.497901 9757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:50.498051 9757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:50.505652 9757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:50.505801 9757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:50.505961 9757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:54:50.516287 9757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:34637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:39961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:50.517659 9757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:50.518923 9757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:50.532179 9762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:50.532138 9763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:50.532203 9765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:50.533321 9757 server_base.cc:1034] running on GCE node
I20260806 10:54:50.534058 9757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:50.535141 9757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:50.536355 9757 hybrid_clock.cc:648] HybridClock initialized: now 1786013690536308 us; error 49 us; skew 500 ppm
I20260806 10:54:50.536676 9757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:50 dist-test-slave-49h3 krb5kdc[9650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013690, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:50.540601 9757 init.cc:377] Logged in from keytab as kudu/127.30.218.1@KRBTEST.COM (short username kudu)
I20260806 10:54:50.542357 9757 webserver.cc:460] Webserver started at http://127.30.218.1:40029/ using document root <none> and password file <none>
I20260806 10:54:50.543134 9757 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:50.543282 9757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:50.543601 9757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:50.546906 9757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "5492f2b7f2b84db69e430302ba071306"
format_stamp: "Formatted at 2026-08-06 10:54:50 on dist-test-slave-49h3"
server_key: "f67d814e091593b36bf1e6b686f5fa84"
server_key_iv: "9c071526dfadeddd137bbe455992b7ef"
server_key_version: "encryptionkey@0"
I20260806 10:54:50.547744 9757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "5492f2b7f2b84db69e430302ba071306"
format_stamp: "Formatted at 2026-08-06 10:54:50 on dist-test-slave-49h3"
server_key: "f67d814e091593b36bf1e6b686f5fa84"
server_key_iv: "9c071526dfadeddd137bbe455992b7ef"
server_key_version: "encryptionkey@0"
I20260806 10:54:50.553535 9757 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260806 10:54:50.557353 9772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:50.558909 9757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:54:50.559103 9757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "5492f2b7f2b84db69e430302ba071306"
format_stamp: "Formatted at 2026-08-06 10:54:50 on dist-test-slave-49h3"
server_key: "f67d814e091593b36bf1e6b686f5fa84"
server_key_iv: "9c071526dfadeddd137bbe455992b7ef"
server_key_version: "encryptionkey@0"
I20260806 10:54:50.559341 9757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:50.583210 9757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:50.587889 9757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:50.588233 9757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:50.589252 9757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:54:50.590773 9757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:50.590858 9757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:50.590991 9757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:50.591046 9757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:50.623420 9757 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:33139
I20260806 10:54:50.623471 9885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:33139 every 8 connection(s)
I20260806 10:54:50.624871 9757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:54:50.625301 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 9757
I20260806 10:54:50.625530 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:54:50.627602 31592 external_mini_cluster.cc:1428] Setting key dc57ab64233fb99941dbcc9cacdfd0ae
Aug 06 10:54:50 dist-test-slave-49h3 krb5kdc[9650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013690, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:54:50.648392 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:50.628803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:46899 (local address 127.30.218.62:39961)
0806 10:54:50.629378 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:50.629385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:50.630607 (+ 1222us) server_negotiation.cc:407] Connection header received
0806 10:54:50.632675 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:50.632682 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:50.632785 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:50.632884 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:50.634749 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:50.635556 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:50.637202 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:50.637664 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:50.637790 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:50.641743 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:50.641749 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:50.641753 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:50.641792 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:50.644968 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:50.645876 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:50.645882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:50.645886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:50.646062 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:50.646489 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:50.646494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:50.646498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:50.646885 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:50.647045 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:50.647744 (+ 699us) server_negotiation.cc:299] Negotiation successful
0806 10:54:50.647986 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":275,"threads_started":1}
I20260806 10:54:50.649199 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:50.629431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39961 (local address 127.30.218.1:46899)
0806 10:54:50.630427 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:50.630478 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:50.632286 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:50.633184 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:50.633194 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:50.633888 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:50.634554 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:50.634579 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:50.635718 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:50.635722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:50.637034 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:50.637049 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:50.637926 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:50.637934 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:50.638184 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:50.638887 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:50.638912 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:50.641418 (+ 2506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:50.645147 (+ 3729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:50.645156 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:50.645172 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:50.645736 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:50.646162 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:50.646165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:50.646167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:50.646360 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:50.646996 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:50.647002 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:50.647355 (+ 353us) client_negotiation.cc:769] Sending connection context
0806 10:54:50.647588 (+ 233us) client_negotiation.cc:240] Negotiation successful
0806 10:54:50.648021 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":367,"threads_started":1}
I20260806 10:54:50.650868 9886 heartbeater.cc:347] Connected to a master server at 127.30.218.62:39961
I20260806 10:54:50.651288 9886 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:50.652216 9886 heartbeater.cc:511] Master 127.30.218.62:39961 requested a full tablet report, sending...
I20260806 10:54:50.654662 9696 ts_manager.cc:194] Registered new tserver with Master: 5492f2b7f2b84db69e430302ba071306 (127.30.218.1:33139)
I20260806 10:54:50.656167 9696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.1@KRBTEST.COM'} at 127.30.218.1:46899
WARNING: no policy specified for kudu/127.30.218.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
I20260806 10:54:50.698899 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:39961
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:34637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:54:51.043476 9892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:51.043823 9892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:51.043967 9892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:51.051776 9892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:51.051924 9892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:51.052084 9892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:54:51.062320 9892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:34637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:39961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:51.063861 9892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:51.065294 9892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:51.080044 9901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:51.080109 9899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:51.080116 9898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:51.081261 9892 server_base.cc:1034] running on GCE node
I20260806 10:54:51.081977 9892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:51.082986 9892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:51.084195 9892 hybrid_clock.cc:648] HybridClock initialized: now 1786013691084144 us; error 44 us; skew 500 ppm
I20260806 10:54:51.084514 9892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:51 dist-test-slave-49h3 krb5kdc[9650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013691, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:51.087666 9892 init.cc:377] Logged in from keytab as kudu/127.30.218.2@KRBTEST.COM (short username kudu)
I20260806 10:54:51.089174 9892 webserver.cc:460] Webserver started at http://127.30.218.2:37063/ using document root <none> and password file <none>
I20260806 10:54:51.089946 9892 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:51.090068 9892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:51.090366 9892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:51.093380 9892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "69972b56bda24be18cbf91db69a56db6"
format_stamp: "Formatted at 2026-08-06 10:54:51 on dist-test-slave-49h3"
server_key: "90e8a453005f377aee0b0ae0b6f200f7"
server_key_iv: "dcdd9d557abc5b204fc9a7e3efb00ce9"
server_key_version: "encryptionkey@0"
I20260806 10:54:51.094219 9892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "69972b56bda24be18cbf91db69a56db6"
format_stamp: "Formatted at 2026-08-06 10:54:51 on dist-test-slave-49h3"
server_key: "90e8a453005f377aee0b0ae0b6f200f7"
server_key_iv: "dcdd9d557abc5b204fc9a7e3efb00ce9"
server_key_version: "encryptionkey@0"
I20260806 10:54:51.100503 9892 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 10:54:51.104169 9908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:51.105654 9892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:54:51.105816 9892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "69972b56bda24be18cbf91db69a56db6"
format_stamp: "Formatted at 2026-08-06 10:54:51 on dist-test-slave-49h3"
server_key: "90e8a453005f377aee0b0ae0b6f200f7"
server_key_iv: "dcdd9d557abc5b204fc9a7e3efb00ce9"
server_key_version: "encryptionkey@0"
I20260806 10:54:51.106012 9892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:51.121402 9892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:51.125955 9892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:51.126302 9892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:51.127470 9892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:54:51.129081 9892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:51.129168 9892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:51.129318 9892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:51.129377 9892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:51.162149 9892 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:38429
I20260806 10:54:51.162214 10021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:38429 every 8 connection(s)
I20260806 10:54:51.163499 9892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:54:51.170094 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 9892
I20260806 10:54:51.170281 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:54:51.170743 31592 external_mini_cluster.cc:1428] Setting key bac28e792a751d50c42120ca9cd82add
Aug 06 10:54:51 dist-test-slave-49h3 krb5kdc[9650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013691, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:54:51.186525 10024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:51.166055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:40097 (local address 127.30.218.62:39961)
0806 10:54:51.166463 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:51.166468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:51.167429 (+ 961us) server_negotiation.cc:407] Connection header received
0806 10:54:51.169172 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:51.169176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:51.169258 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:51.169353 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:51.170978 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.171927 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.174194 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.174746 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.174884 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:51.178740 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:51.178747 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:51.178751 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:51.178799 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:51.182649 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:51.183707 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:51.183714 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:51.183717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:51.183902 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:51.184475 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:51.184479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:51.184482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:51.184873 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:51.185024 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:51.185896 (+ 872us) server_negotiation.cc:299] Negotiation successful
0806 10:54:51.186176 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":200,"threads_started":1}
I20260806 10:54:51.187463 10025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:51.166569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39961 (local address 127.30.218.2:40097)
0806 10:54:51.167261 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:51.167305 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:51.168926 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:51.169568 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:51.169577 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:51.170089 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:51.170736 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:51.170756 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.172099 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.172104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:51.174012 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:51.174028 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.175002 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.175008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:51.175246 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:51.175966 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:51.175991 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:51.178414 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:51.182922 (+ 4508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:51.182931 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:51.182949 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:51.183523 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:51.184060 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:51.184064 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:51.184067 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:51.184341 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:51.184994 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:51.185001 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:51.185421 (+ 420us) client_negotiation.cc:769] Sending connection context
0806 10:54:51.185734 (+ 313us) client_negotiation.cc:240] Negotiation successful
0806 10:54:51.186228 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":261,"threads_started":1}
I20260806 10:54:51.189077 10022 heartbeater.cc:347] Connected to a master server at 127.30.218.62:39961
I20260806 10:54:51.189484 10022 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:51.190451 10022 heartbeater.cc:511] Master 127.30.218.62:39961 requested a full tablet report, sending...
I20260806 10:54:51.192312 9696 ts_manager.cc:194] Registered new tserver with Master: 69972b56bda24be18cbf91db69a56db6 (127.30.218.2:38429)
I20260806 10:54:51.193222 9696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.2@KRBTEST.COM'} at 127.30.218.2:40097
WARNING: no policy specified for kudu/127.30.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
I20260806 10:54:51.245154 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:39961
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:34637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:54:51.599431 10028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:54:51.599742 10028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:54:51.599882 10028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:54:51.607662 10028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:54:51.607797 10028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:54:51.607959 10028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:54:51.618561 10028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:34637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:39961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:54:51.619964 10028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:54:51.621361 10028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:54:51.635078 10036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:51.635130 10034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:54:51.635159 10033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:54:51.636258 10028 server_base.cc:1034] running on GCE node
I20260806 10:54:51.636960 10028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:54:51.638047 10028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:54:51.639254 10028 hybrid_clock.cc:648] HybridClock initialized: now 1786013691639203 us; error 43 us; skew 500 ppm
I20260806 10:54:51.639587 10028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:54:51 dist-test-slave-49h3 krb5kdc[9650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013691, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:54:51.643003 10028 init.cc:377] Logged in from keytab as kudu/127.30.218.3@KRBTEST.COM (short username kudu)
I20260806 10:54:51.644618 10028 webserver.cc:460] Webserver started at http://127.30.218.3:45979/ using document root <none> and password file <none>
I20260806 10:54:51.645391 10028 fs_manager.cc:359] Metadata directory not provided
I20260806 10:54:51.645558 10028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:54:51.645913 10028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:54:51.649217 10028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "fbac60810721408caf1a1a3a62f33bf2"
format_stamp: "Formatted at 2026-08-06 10:54:51 on dist-test-slave-49h3"
server_key: "557452c940f1b579b99292bdff675f72"
server_key_iv: "cf0628b502a3141cbb47e0a73a23250b"
server_key_version: "encryptionkey@0"
I20260806 10:54:51.650197 10028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "fbac60810721408caf1a1a3a62f33bf2"
format_stamp: "Formatted at 2026-08-06 10:54:51 on dist-test-slave-49h3"
server_key: "557452c940f1b579b99292bdff675f72"
server_key_iv: "cf0628b502a3141cbb47e0a73a23250b"
server_key_version: "encryptionkey@0"
I20260806 10:54:51.656086 10028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:54:51.660100 10043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:51.660284 9886 heartbeater.cc:503] Master 127.30.218.62:39961 was elected leader, sending a full tablet report...
I20260806 10:54:51.662012 10028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 10:54:51.662216 10028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "fbac60810721408caf1a1a3a62f33bf2"
format_stamp: "Formatted at 2026-08-06 10:54:51 on dist-test-slave-49h3"
server_key: "557452c940f1b579b99292bdff675f72"
server_key_iv: "cf0628b502a3141cbb47e0a73a23250b"
server_key_version: "encryptionkey@0"
I20260806 10:54:51.662487 10028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:54:51.680655 10028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:54:51.685346 10028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:54:51.685715 10028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:54:51.686839 10028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:54:51.688365 10028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:54:51.688443 10028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:51.688545 10028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:54:51.688589 10028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:54:51.721035 10028 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:45213
I20260806 10:54:51.721077 10156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:45213 every 8 connection(s)
I20260806 10:54:51.722579 10028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:54:51.727365 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 10028
I20260806 10:54:51.727569 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:54:51.728124 31592 external_mini_cluster.cc:1428] Setting key 7f5e78e36adb9f5393b8b897d54d7558
Aug 06 10:54:51 dist-test-slave-49h3 krb5kdc[9650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013691, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:54:51.755347 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:51.725480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:47437 (local address 127.30.218.62:39961)
0806 10:54:51.725948 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:51.725956 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:51.737803 (+ 11847us) server_negotiation.cc:407] Connection header received
0806 10:54:51.739884 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:51.739889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:51.739984 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:51.740088 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:51.741911 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.742755 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.744492 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.744930 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.745050 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:51.748874 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:51.748879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:51.748883 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:51.748922 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:51.751865 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:51.752755 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:51.752760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:51.752762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:51.752909 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:51.753373 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:51.753377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:51.753380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:51.753735 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:51.753902 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:51.754676 (+ 774us) server_negotiation.cc:299] Negotiation successful
0806 10:54:51.754939 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":230,"threads_started":1}
I20260806 10:54:51.756371 10160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:51.726076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:39961 (local address 127.30.218.3:47437)
0806 10:54:51.737564 (+ 11488us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:51.737651 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:51.739534 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:51.740339 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:51.740349 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:51.740958 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:54:51.741673 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:51.741697 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.742929 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.742933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:51.744328 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:51.744342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.745178 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.745186 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:51.745435 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:51.746177 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:54:51.746203 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:54:51.748606 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:54:51.752068 (+ 3462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:51.752079 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:51.752100 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:51.752605 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:51.753025 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:54:51.753030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:54:51.753034 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:54:51.753243 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:54:51.753880 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:54:51.753888 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:54:51.754266 (+ 378us) client_negotiation.cc:769] Sending connection context
0806 10:54:51.754529 (+ 263us) client_negotiation.cc:240] Negotiation successful
0806 10:54:51.755141 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11064,"thread_start_us":10838,"threads_started":1}
I20260806 10:54:51.757896 10157 heartbeater.cc:347] Connected to a master server at 127.30.218.62:39961
I20260806 10:54:51.758172 10157 heartbeater.cc:464] Registering TS with master...
I20260806 10:54:51.758919 10157 heartbeater.cc:511] Master 127.30.218.62:39961 requested a full tablet report, sending...
I20260806 10:54:51.760298 9696 ts_manager.cc:194] Registered new tserver with Master: fbac60810721408caf1a1a3a62f33bf2 (127.30.218.3:45213)
I20260806 10:54:51.761129 9696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.3@KRBTEST.COM'} at 127.30.218.3:47437
I20260806 10:54:51.770519 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:54:51.787743 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:51.775643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43994 (local address 127.30.218.62:39961)
0806 10:54:51.775842 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:51.775846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:51.776328 (+ 482us) server_negotiation.cc:407] Connection header received
0806 10:54:51.776461 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:51.776465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:51.776537 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:51.776625 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:51.778126 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.778855 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.780013 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.780399 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.780488 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:51.782815 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:51.782820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:51.782824 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:51.782859 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:51.785407 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:51.786069 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:51.786072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:51.786075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:51.786197 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:51.786504 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:51.786507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:51.786509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:51.786851 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:51.787050 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:51.787311 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 10:54:51.787492 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 10:54:51.792006 9696 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:54:51.799190 9696 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:54:51.833348 10171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:51.812155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:33139 (local address 127.0.0.1:59634)
0806 10:54:51.822570 (+ 10415us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:51.822612 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:51.822812 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:51.824112 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:51.824121 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:51.824158 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:51.824626 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:51.824639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.827020 (+ 2381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.827027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:51.830260 (+ 3233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:51.830276 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.832183 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.832191 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:51.832458 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:51.832519 (+ 61us) client_negotiation.cc:769] Sending connection context
0806 10:54:51.832635 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 10:54:51.832857 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10135,"thread_start_us":10834,"threads_started":1}
I20260806 10:54:51.833725 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:51.812651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59634 (local address 127.30.218.1:33139)
0806 10:54:51.823410 (+ 10759us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:51.823420 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:51.823454 (+ 34us) server_negotiation.cc:407] Connection header received
0806 10:54:51.823575 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:51.823584 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:51.823856 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:51.824065 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:51.824903 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.826792 (+ 1889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.830473 (+ 3681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.831936 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.832721 (+ 785us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:51.833097 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:51.833177 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 10:54:51.833375 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10557,"thread_start_us":10362,"threads_started":1}
I20260806 10:54:51.837699 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:51.812380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:45213 (local address 127.0.0.1:40046)
0806 10:54:51.827889 (+ 15509us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:51.827911 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:51.828053 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:51.830439 (+ 2386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:51.830445 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:51.830476 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:51.830910 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:51.830922 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.832862 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.832867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:51.835374 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:51.835392 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.836742 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.836749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:51.837000 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:51.837027 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 10:54:51.837118 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 10:54:51.837281 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15350,"mutex_wait_us":47,"thread_start_us":16188,"threads_started":1}
I20260806 10:54:51.838117 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:51.812932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40046 (local address 127.30.218.3:45213)
0806 10:54:51.829383 (+ 16451us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:51.829392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:51.829432 (+ 40us) server_negotiation.cc:407] Connection header received
0806 10:54:51.829573 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:51.829581 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:51.829881 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:51.830117 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:51.831122 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.832716 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.835539 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.836515 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.836647 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:51.836892 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:51.837210 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 10:54:51.837398 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16269,"thread_start_us":16068,"threads_started":1}
I20260806 10:54:51.838223 9821 tablet_service.cc:1467] Processing CreateTablet for tablet 13578c2e62134b9bb79d04e056c45bb0 (DEFAULT_TABLE table=test-table [id=f844ebc2d42c4d39aabb803b33e5461b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:54:51.839838 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:51.815197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:38429 (local address 127.0.0.1:57026)
0806 10:54:51.828858 (+ 13661us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:51.828879 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:51.828995 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:51.832041 (+ 3046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:51.832046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:51.832071 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:51.832497 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:51.832506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.834835 (+ 2329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.834841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:51.836968 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:51.836985 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.838990 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.838996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:51.839177 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:51.839216 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 10:54:51.839302 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 10:54:51.839429 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13537,"mutex_wait_us":6,"thread_start_us":12941,"threads_started":1}
I20260806 10:54:51.839870 9821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13578c2e62134b9bb79d04e056c45bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:51.840507 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:51.815387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57026 (local address 127.30.218.2:38429)
0806 10:54:51.831496 (+ 16109us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:51.831505 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:51.831541 (+ 36us) server_negotiation.cc:407] Connection header received
0806 10:54:51.831622 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:51.831627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:51.831833 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:51.832010 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:51.832672 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.834643 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.837151 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.838476 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.838636 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:51.838836 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:51.840073 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0806 10:54:51.840263 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15961,"thread_start_us":15821,"threads_started":1}
I20260806 10:54:51.842079 10091 tablet_service.cc:1467] Processing CreateTablet for tablet 13578c2e62134b9bb79d04e056c45bb0 (DEFAULT_TABLE table=test-table [id=f844ebc2d42c4d39aabb803b33e5461b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:54:51.843550 10091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13578c2e62134b9bb79d04e056c45bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:51.844237 9956 tablet_service.cc:1467] Processing CreateTablet for tablet 13578c2e62134b9bb79d04e056c45bb0 (DEFAULT_TABLE table=test-table [id=f844ebc2d42c4d39aabb803b33e5461b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:54:51.845323 9956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13578c2e62134b9bb79d04e056c45bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:54:51.856046 10177 tablet_bootstrap.cc:492] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306: Bootstrap starting.
I20260806 10:54:51.857827 10178 tablet_bootstrap.cc:492] T 13578c2e62134b9bb79d04e056c45bb0 P 69972b56bda24be18cbf91db69a56db6: Bootstrap starting.
I20260806 10:54:51.859844 10177 tablet_bootstrap.cc:654] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:51.860812 10179 tablet_bootstrap.cc:492] T 13578c2e62134b9bb79d04e056c45bb0 P fbac60810721408caf1a1a3a62f33bf2: Bootstrap starting.
I20260806 10:54:51.861224 10177 log.cc:824] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306: Log is configured to *not* fsync() on all Append() calls
I20260806 10:54:51.861742 10178 tablet_bootstrap.cc:654] T 13578c2e62134b9bb79d04e056c45bb0 P 69972b56bda24be18cbf91db69a56db6: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:51.863108 10178 log.cc:824] T 13578c2e62134b9bb79d04e056c45bb0 P 69972b56bda24be18cbf91db69a56db6: Log is configured to *not* fsync() on all Append() calls
I20260806 10:54:51.864835 10177 tablet_bootstrap.cc:492] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306: No bootstrap required, opened a new log
I20260806 10:54:51.864910 10179 tablet_bootstrap.cc:654] T 13578c2e62134b9bb79d04e056c45bb0 P fbac60810721408caf1a1a3a62f33bf2: Neither blocks nor log segments found. Creating new log.
I20260806 10:54:51.865132 10177 ts_tablet_manager.cc:1397] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260806 10:54:51.866067 10179 log.cc:824] T 13578c2e62134b9bb79d04e056c45bb0 P fbac60810721408caf1a1a3a62f33bf2: Log is configured to *not* fsync() on all Append() calls
I20260806 10:54:51.867036 10178 tablet_bootstrap.cc:492] T 13578c2e62134b9bb79d04e056c45bb0 P 69972b56bda24be18cbf91db69a56db6: No bootstrap required, opened a new log
I20260806 10:54:51.867336 10178 ts_tablet_manager.cc:1397] T 13578c2e62134b9bb79d04e056c45bb0 P 69972b56bda24be18cbf91db69a56db6: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260806 10:54:51.868830 10179 tablet_bootstrap.cc:492] T 13578c2e62134b9bb79d04e056c45bb0 P fbac60810721408caf1a1a3a62f33bf2: No bootstrap required, opened a new log
I20260806 10:54:51.869073 10179 ts_tablet_manager.cc:1397] T 13578c2e62134b9bb79d04e056c45bb0 P fbac60810721408caf1a1a3a62f33bf2: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 10:54:51.876492 10177 raft_consensus.cc:348] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5492f2b7f2b84db69e430302ba071306" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33139 } } peers { permanent_uuid: "fbac60810721408caf1a1a3a62f33bf2" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45213 } } peers { permanent_uuid: "69972b56bda24be18cbf91db69a56db6" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 38429 } }
I20260806 10:54:51.877058 10177 raft_consensus.cc:374] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:51.877166 10177 raft_consensus.cc:729] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5492f2b7f2b84db69e430302ba071306, State: Initialized, Role: FOLLOWER
I20260806 10:54:51.877046 10179 raft_consensus.cc:348] T 13578c2e62134b9bb79d04e056c45bb0 P fbac60810721408caf1a1a3a62f33bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5492f2b7f2b84db69e430302ba071306" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33139 } } peers { permanent_uuid: "fbac60810721408caf1a1a3a62f33bf2" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45213 } } peers { permanent_uuid: "69972b56bda24be18cbf91db69a56db6" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 38429 } }
I20260806 10:54:51.877540 10179 raft_consensus.cc:374] T 13578c2e62134b9bb79d04e056c45bb0 P fbac60810721408caf1a1a3a62f33bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:51.877646 10179 raft_consensus.cc:729] T 13578c2e62134b9bb79d04e056c45bb0 P fbac60810721408caf1a1a3a62f33bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbac60810721408caf1a1a3a62f33bf2, State: Initialized, Role: FOLLOWER
I20260806 10:54:51.878029 10177 consensus_queue.cc:260] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5492f2b7f2b84db69e430302ba071306" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33139 } } peers { permanent_uuid: "fbac60810721408caf1a1a3a62f33bf2" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45213 } } peers { permanent_uuid: "69972b56bda24be18cbf91db69a56db6" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 38429 } }
I20260806 10:54:51.878062 10178 raft_consensus.cc:348] T 13578c2e62134b9bb79d04e056c45bb0 P 69972b56bda24be18cbf91db69a56db6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5492f2b7f2b84db69e430302ba071306" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33139 } } peers { permanent_uuid: "fbac60810721408caf1a1a3a62f33bf2" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45213 } } peers { permanent_uuid: "69972b56bda24be18cbf91db69a56db6" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 38429 } }
I20260806 10:54:51.878207 10179 consensus_queue.cc:260] T 13578c2e62134b9bb79d04e056c45bb0 P fbac60810721408caf1a1a3a62f33bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5492f2b7f2b84db69e430302ba071306" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33139 } } peers { permanent_uuid: "fbac60810721408caf1a1a3a62f33bf2" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45213 } } peers { permanent_uuid: "69972b56bda24be18cbf91db69a56db6" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 38429 } }
I20260806 10:54:51.878470 10178 raft_consensus.cc:374] T 13578c2e62134b9bb79d04e056c45bb0 P 69972b56bda24be18cbf91db69a56db6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:54:51.878563 10178 raft_consensus.cc:729] T 13578c2e62134b9bb79d04e056c45bb0 P 69972b56bda24be18cbf91db69a56db6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69972b56bda24be18cbf91db69a56db6, State: Initialized, Role: FOLLOWER
I20260806 10:54:51.879446 10157 heartbeater.cc:503] Master 127.30.218.62:39961 was elected leader, sending a full tablet report...
I20260806 10:54:51.880049 10179 ts_tablet_manager.cc:1428] T 13578c2e62134b9bb79d04e056c45bb0 P fbac60810721408caf1a1a3a62f33bf2: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260806 10:54:51.881055 10178 consensus_queue.cc:260] T 13578c2e62134b9bb79d04e056c45bb0 P 69972b56bda24be18cbf91db69a56db6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5492f2b7f2b84db69e430302ba071306" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33139 } } peers { permanent_uuid: "fbac60810721408caf1a1a3a62f33bf2" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45213 } } peers { permanent_uuid: "69972b56bda24be18cbf91db69a56db6" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 38429 } }
I20260806 10:54:51.882416 10022 heartbeater.cc:503] Master 127.30.218.62:39961 was elected leader, sending a full tablet report...
I20260806 10:54:51.883137 10178 ts_tablet_manager.cc:1428] T 13578c2e62134b9bb79d04e056c45bb0 P 69972b56bda24be18cbf91db69a56db6: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260806 10:54:51.884832 10177 ts_tablet_manager.cc:1428] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306: Time spent starting tablet: real 0.020s user 0.007s sys 0.011s
W20260806 10:54:51.915570 10023 tablet.cc:2369] T 13578c2e62134b9bb79d04e056c45bb0 P 69972b56bda24be18cbf91db69a56db6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:54:51.974427 10158 tablet.cc:2369] T 13578c2e62134b9bb79d04e056c45bb0 P fbac60810721408caf1a1a3a62f33bf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:54:51.990422 10183 raft_consensus.cc:482] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:54:51.990753 10183 raft_consensus.cc:504] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5492f2b7f2b84db69e430302ba071306" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33139 } } peers { permanent_uuid: "fbac60810721408caf1a1a3a62f33bf2" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45213 } } peers { permanent_uuid: "69972b56bda24be18cbf91db69a56db6" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 38429 } }
I20260806 10:54:51.992447 10183 leader_election.cc:288] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbac60810721408caf1a1a3a62f33bf2 (127.30.218.3:45213), 69972b56bda24be18cbf91db69a56db6 (127.30.218.2:38429)
I20260806 10:54:52.001580 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:51.992944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:39341 (local address 127.30.218.3:45213)
0806 10:54:51.993159 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:51.993167 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:51.994309 (+ 1142us) server_negotiation.cc:407] Connection header received
0806 10:54:51.994627 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:51.994633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:51.994713 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:51.994795 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:51.995459 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.996550 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.999364 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:52.000541 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:52.000705 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:52.000776 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:52.001181 (+ 405us) server_negotiation.cc:299] Negotiation successful
0806 10:54:52.001311 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 10:54:52.001592 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:51.993477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:36017 (local address 127.30.218.2:38429)
0806 10:54:51.993674 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:51.993680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:51.994619 (+ 939us) server_negotiation.cc:407] Connection header received
0806 10:54:51.994765 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:51.994770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:51.994844 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:51.994931 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:54:51.995600 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.996611 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.999364 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:52.000543 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:52.000703 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:52.000776 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:52.001181 (+ 405us) server_negotiation.cc:299] Negotiation successful
0806 10:54:52.001311 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 10:54:52.001821 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:51.992929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:38429 (local address 127.30.218.1:36017)
0806 10:54:51.994523 (+ 1594us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:51.994539 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:51.994644 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:51.994980 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:51.994985 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:51.995037 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:51.995435 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:51.995448 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.996744 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.996748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:51.999199 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:51.999214 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:52.000800 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:52.000806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:52.000980 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:52.001060 (+ 80us) client_negotiation.cc:769] Sending connection context
0806 10:54:52.001137 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 10:54:52.001295 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1485,"mutex_wait_us":176,"thread_start_us":1292,"threads_started":1}
I20260806 10:54:52.001840 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:51.992868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:45213 (local address 127.30.218.1:39341)
0806 10:54:51.994192 (+ 1324us) negotiation.cc:107] Waiting for socket to connect
0806 10:54:51.994215 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:54:51.994438 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:54:51.994822 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:54:51.994827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:54:51.994858 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:54:51.995307 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:51.995318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:51.996726 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:51.996731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:51.999199 (+ 2468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:54:51.999214 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:54:52.000762 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:54:52.000769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:54:52.000981 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:52.001046 (+ 65us) client_negotiation.cc:769] Sending connection context
0806 10:54:52.001137 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 10:54:52.001291 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1195,"mutex_wait_us":134,"spinlock_wait_cycles":22912,"thread_start_us":995,"threads_started":1}
I20260806 10:54:52.002744 9976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13578c2e62134b9bb79d04e056c45bb0" candidate_uuid: "5492f2b7f2b84db69e430302ba071306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69972b56bda24be18cbf91db69a56db6" is_pre_election: true
I20260806 10:54:52.002770 10111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13578c2e62134b9bb79d04e056c45bb0" candidate_uuid: "5492f2b7f2b84db69e430302ba071306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbac60810721408caf1a1a3a62f33bf2" is_pre_election: true
I20260806 10:54:52.003233 10111 raft_consensus.cc:2454] T 13578c2e62134b9bb79d04e056c45bb0 P fbac60810721408caf1a1a3a62f33bf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5492f2b7f2b84db69e430302ba071306 in term 0.
I20260806 10:54:52.003222 9976 raft_consensus.cc:2454] T 13578c2e62134b9bb79d04e056c45bb0 P 69972b56bda24be18cbf91db69a56db6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5492f2b7f2b84db69e430302ba071306 in term 0.
I20260806 10:54:52.003840 9777 leader_election.cc:302] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5492f2b7f2b84db69e430302ba071306, 69972b56bda24be18cbf91db69a56db6, fbac60810721408caf1a1a3a62f33bf2; no voters:
I20260806 10:54:52.004184 10183 raft_consensus.cc:2781] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:54:52.004331 10183 raft_consensus.cc:482] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:54:52.004442 10183 raft_consensus.cc:3037] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:52.006681 10183 raft_consensus.cc:504] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5492f2b7f2b84db69e430302ba071306" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33139 } } peers { permanent_uuid: "fbac60810721408caf1a1a3a62f33bf2" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45213 } } peers { permanent_uuid: "69972b56bda24be18cbf91db69a56db6" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 38429 } }
I20260806 10:54:52.007282 10183 leader_election.cc:288] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306 [CANDIDATE]: Term 1 election: Requested vote from peers fbac60810721408caf1a1a3a62f33bf2 (127.30.218.3:45213), 69972b56bda24be18cbf91db69a56db6 (127.30.218.2:38429)
I20260806 10:54:52.007732 10111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13578c2e62134b9bb79d04e056c45bb0" candidate_uuid: "5492f2b7f2b84db69e430302ba071306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbac60810721408caf1a1a3a62f33bf2"
I20260806 10:54:52.007741 9976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13578c2e62134b9bb79d04e056c45bb0" candidate_uuid: "5492f2b7f2b84db69e430302ba071306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69972b56bda24be18cbf91db69a56db6"
I20260806 10:54:52.007921 9976 raft_consensus.cc:3037] T 13578c2e62134b9bb79d04e056c45bb0 P 69972b56bda24be18cbf91db69a56db6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:52.007923 10111 raft_consensus.cc:3037] T 13578c2e62134b9bb79d04e056c45bb0 P fbac60810721408caf1a1a3a62f33bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:54:52.010255 10111 raft_consensus.cc:2454] T 13578c2e62134b9bb79d04e056c45bb0 P fbac60810721408caf1a1a3a62f33bf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5492f2b7f2b84db69e430302ba071306 in term 1.
I20260806 10:54:52.010339 9976 raft_consensus.cc:2454] T 13578c2e62134b9bb79d04e056c45bb0 P 69972b56bda24be18cbf91db69a56db6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5492f2b7f2b84db69e430302ba071306 in term 1.
I20260806 10:54:52.010768 9777 leader_election.cc:302] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5492f2b7f2b84db69e430302ba071306, 69972b56bda24be18cbf91db69a56db6, fbac60810721408caf1a1a3a62f33bf2; no voters:
I20260806 10:54:52.011054 10183 raft_consensus.cc:2781] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:54:52.011494 10183 raft_consensus.cc:686] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306 [term 1 LEADER]: Becoming Leader. State: Replica: 5492f2b7f2b84db69e430302ba071306, State: Running, Role: LEADER
I20260806 10:54:52.011976 10183 consensus_queue.cc:237] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5492f2b7f2b84db69e430302ba071306" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33139 } } peers { permanent_uuid: "fbac60810721408caf1a1a3a62f33bf2" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45213 } } peers { permanent_uuid: "69972b56bda24be18cbf91db69a56db6" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 38429 } }
I20260806 10:54:52.016739 9696 catalog_manager.cc:5495] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5492f2b7f2b84db69e430302ba071306 (127.30.218.1). New cstate: current_term: 1 leader_uuid: "5492f2b7f2b84db69e430302ba071306" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5492f2b7f2b84db69e430302ba071306" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbac60810721408caf1a1a3a62f33bf2" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69972b56bda24be18cbf91db69a56db6" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 38429 } health_report { overall_health: UNKNOWN } } }
I20260806 10:54:52.037405 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:52.032182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59650 (local address 127.30.218.1:33139)
0806 10:54:52.032354 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:52.032358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:52.032376 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:54:52.032556 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:52.032560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:52.032638 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:52.032786 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:54:52.033183 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:52.033984 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:52.035283 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:52.035695 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:52.035790 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:52.036334 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:54:52.036882 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:54:52.036992 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:52.037099 (+ 107us) server_negotiation.cc:299] Negotiation successful
0806 10:54:52.037176 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 10:54:52.045958 9976 raft_consensus.cc:1261] T 13578c2e62134b9bb79d04e056c45bb0 P 69972b56bda24be18cbf91db69a56db6 [term 1 FOLLOWER]: Refusing update from remote peer 5492f2b7f2b84db69e430302ba071306: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:54:52.045958 10111 raft_consensus.cc:1261] T 13578c2e62134b9bb79d04e056c45bb0 P fbac60810721408caf1a1a3a62f33bf2 [term 1 FOLLOWER]: Refusing update from remote peer 5492f2b7f2b84db69e430302ba071306: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:54:52.047030 10183 consensus_queue.cc:1035] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbac60810721408caf1a1a3a62f33bf2" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 45213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:54:52.047358 10188 consensus_queue.cc:1035] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69972b56bda24be18cbf91db69a56db6" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 38429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:54:52.057427 10191 mvcc.cc:205] Tried to move back new op lower bound from 7315512082609803264 to 7315512082484936704. Current Snapshot: MvccSnapshot[applied={T|T < 7315512082609803264}]
I20260806 10:54:52.074088 9695 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:54:52.074457 9695 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:54:52.078450 9695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89172c9a7ac6402cacb26270870be367: Sending DeleteTablet for 3 replicas of tablet 13578c2e62134b9bb79d04e056c45bb0
I20260806 10:54:52.079725 10091 tablet_service.cc:1514] Processing DeleteTablet for tablet 13578c2e62134b9bb79d04e056c45bb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:54:52 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:40046
I20260806 10:54:52.079732 9821 tablet_service.cc:1514] Processing DeleteTablet for tablet 13578c2e62134b9bb79d04e056c45bb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:54:52 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:59634
I20260806 10:54:52.080281 9956 tablet_service.cc:1514] Processing DeleteTablet for tablet 13578c2e62134b9bb79d04e056c45bb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:54:52 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:57026
I20260806 10:54:52.082643 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 9757
I20260806 10:54:52.093263 10202 tablet_replica.cc:330] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306: stopping tablet replica
I20260806 10:54:52.093982 10201 tablet_replica.cc:330] T 13578c2e62134b9bb79d04e056c45bb0 P fbac60810721408caf1a1a3a62f33bf2: stopping tablet replica
I20260806 10:54:52.094044 10203 tablet_replica.cc:330] T 13578c2e62134b9bb79d04e056c45bb0 P 69972b56bda24be18cbf91db69a56db6: stopping tablet replica
I20260806 10:54:52.094581 10203 raft_consensus.cc:2229] T 13578c2e62134b9bb79d04e056c45bb0 P 69972b56bda24be18cbf91db69a56db6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:54:52.094583 10201 raft_consensus.cc:2229] T 13578c2e62134b9bb79d04e056c45bb0 P fbac60810721408caf1a1a3a62f33bf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:54:52.095151 10203 raft_consensus.cc:2258] T 13578c2e62134b9bb79d04e056c45bb0 P 69972b56bda24be18cbf91db69a56db6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:54:52.095712 10201 raft_consensus.cc:2258] T 13578c2e62134b9bb79d04e056c45bb0 P fbac60810721408caf1a1a3a62f33bf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:54:52.095167 10202 raft_consensus.cc:2229] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:54:52.096542 10202 raft_consensus.cc:2258] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:54:52.099215 10202 ts_tablet_manager.cc:1905] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:54:52.099215 10201 ts_tablet_manager.cc:1905] T 13578c2e62134b9bb79d04e056c45bb0 P fbac60810721408caf1a1a3a62f33bf2: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 10:54:52 dist-test-slave-49h3 krb5kdc[9650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.1@KRBTEST.COM
I20260806 10:54:52.107127 10201 ts_tablet_manager.cc:1918] T 13578c2e62134b9bb79d04e056c45bb0 P fbac60810721408caf1a1a3a62f33bf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:54:52.107362 10201 log.cc:1196] T 13578c2e62134b9bb79d04e056c45bb0 P fbac60810721408caf1a1a3a62f33bf2: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/13578c2e62134b9bb79d04e056c45bb0
I20260806 10:54:52.107501 10202 ts_tablet_manager.cc:1918] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:54:52.107928 10201 ts_tablet_manager.cc:1939] T 13578c2e62134b9bb79d04e056c45bb0 P fbac60810721408caf1a1a3a62f33bf2: Deleting consensus metadata
I20260806 10:54:52.108820 10203 ts_tablet_manager.cc:1905] T 13578c2e62134b9bb79d04e056c45bb0 P 69972b56bda24be18cbf91db69a56db6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:54:52.109695 10202 log.cc:1196] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-0/wal/wals/13578c2e62134b9bb79d04e056c45bb0
I20260806 10:54:52.110152 9681 catalog_manager.cc:4841] TS fbac60810721408caf1a1a3a62f33bf2 (127.30.218.3:45213): tablet 13578c2e62134b9bb79d04e056c45bb0 (table test-table [id=f844ebc2d42c4d39aabb803b33e5461b]) successfully deleted
I20260806 10:54:52.110863 10202 ts_tablet_manager.cc:1939] T 13578c2e62134b9bb79d04e056c45bb0 P 5492f2b7f2b84db69e430302ba071306: Deleting consensus metadata
I20260806 10:54:52.112807 9682 catalog_manager.cc:4841] TS 5492f2b7f2b84db69e430302ba071306 (127.30.218.1:33139): tablet 13578c2e62134b9bb79d04e056c45bb0 (table test-table [id=f844ebc2d42c4d39aabb803b33e5461b]) successfully deleted
I20260806 10:54:52.115693 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:52.084170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.30.218.1:33139)
0806 10:54:52.088249 (+ 4079us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:52.088256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:52.088279 (+ 23us) server_negotiation.cc:407] Connection header received
0806 10:54:52.089262 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:52.089270 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:52.089376 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:52.089495 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:52.092650 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:52.094056 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:52.095945 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:52.096697 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:52.096841 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:52.104076 (+ 7235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:52.104094 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:52.104249 (+ 155us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:52.104314 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:52.109281 (+ 4967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:52.113232 (+ 3951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:52.113241 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:52.113244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:52.113436 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:52.113980 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:52.113984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:52.113986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:52.114384 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:52.114699 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:52.115061 (+ 362us) server_negotiation.cc:299] Negotiation successful
0806 10:54:52.115335 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3925}
I20260806 10:54:52.116420 9881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:52.118153 10203 ts_tablet_manager.cc:1918] T 13578c2e62134b9bb79d04e056c45bb0 P 69972b56bda24be18cbf91db69a56db6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:54:52.118439 10203 log.cc:1196] T 13578c2e62134b9bb79d04e056c45bb0 P 69972b56bda24be18cbf91db69a56db6: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/13578c2e62134b9bb79d04e056c45bb0
I20260806 10:54:52.119030 10203 ts_tablet_manager.cc:1939] T 13578c2e62134b9bb79d04e056c45bb0 P 69972b56bda24be18cbf91db69a56db6: Deleting consensus metadata
I20260806 10:54:52.121100 9683 catalog_manager.cc:4841] TS 69972b56bda24be18cbf91db69a56db6 (127.30.218.2:38429): tablet 13578c2e62134b9bb79d04e056c45bb0 (table test-table [id=f844ebc2d42c4d39aabb803b33e5461b]) successfully deleted
I20260806 10:54:52.216976 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 9757
I20260806 10:54:52.236135 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 9892
Aug 06 10:54:52 dist-test-slave-49h3 krb5kdc[9650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.2@KRBTEST.COM
I20260806 10:54:52.250830 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:52.236985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57036 (local address 127.30.218.2:38429)
0806 10:54:52.237174 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:52.237179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:52.237210 (+ 31us) server_negotiation.cc:407] Connection header received
0806 10:54:52.237282 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:52.237287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:52.237365 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:52.237472 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:52.238736 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:52.239508 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:52.240723 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:52.241115 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:52.241233 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:52.245235 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:52.245248 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:52.245263 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:52.245298 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:52.247963 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:52.248618 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:52.248626 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:52.248629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:52.248760 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:52.249117 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:52.249123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:52.249126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:52.249646 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:52.249941 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:52.250262 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 10:54:52.250489 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 10:54:52.251524 10017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:52.342795 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 9892
I20260806 10:54:52.360292 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 10028
Aug 06 10:54:52 dist-test-slave-49h3 krb5kdc[9650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.3@KRBTEST.COM
I20260806 10:54:52.376713 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:54:52.361221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40048 (local address 127.30.218.3:45213)
0806 10:54:52.361382 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 10:54:52.361387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:54:52.361424 (+ 37us) server_negotiation.cc:407] Connection header received
0806 10:54:52.361504 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:54:52.361508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:54:52.361585 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:54:52.361665 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:54:52.363053 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:52.363850 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:52.365132 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:54:52.365677 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:54:52.365792 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:54:52.370591 (+ 4799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:54:52.370611 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:54:52.370628 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:54:52.370684 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:54:52.373603 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:52.374468 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:52.374479 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:52.374483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:52.374672 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:54:52.375107 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:54:52.375112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:54:52.375114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:54:52.375572 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:54:52.375811 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0806 10:54:52.376157 (+ 346us) server_negotiation.cc:299] Negotiation successful
0806 10:54:52.376377 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 10:54:52.377377 10152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:52.468371 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 10028
I20260806 10:54:52.486816 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 9664
I20260806 10:54:52.487975 9726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:54:52.602975 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 9664
2026-08-06T10:54:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4475 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:54:52 dist-test-slave-49h3 krb5kdc[10211](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:54:52 dist-test-slave-49h3 krb5kdc[10211](info): set up 2 sockets
Aug 06 10:54:52 dist-test-slave-49h3 krb5kdc[10211](info): commencing operation
krb5kdc: starting...
W20260806 10:54:53.678642 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:54:53 dist-test-slave-49h3 krb5kdc[10211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:54:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:54:53Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.30.218.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.30.218.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.30.218.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/rangeradmin_127.30.218.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.30.218.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/rangeradmin_127.30.218.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.30.218.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.30.218.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.30.218.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/rangerlookup_127.30.218.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.30.218.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/rangerlookup_127.30.218.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.218.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.30.218.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.218.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/HTTP_127.30.218.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.218.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/HTTP_127.30.218.20@KRBTEST.COM.keytab.
I20260806 10:54:54.129415 31592 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskT4s7YX/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/postgres -l logfile start
2026-08-06 10:54:56.606 UTC [10241] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 10:54:56.607 UTC [10241] LOG: listening on IPv4 address "127.30.218.20", port 59275
2026-08-06 10:54:56.610 UTC [10241] LOG: listening on Unix socket "/tmp/.s.PGSQL.59275"
2026-08-06 10:54:56.635 UTC [10244] LOG: database system was shut down at 2026-08-06 10:54:55 UTC
2026-08-06 10:54:56.639 UTC [10241] LOG: database system is ready to accept connections
I20260806 10:54:57.585695 31592 mini_postgres.cc:96] Postgres bound to 59275
2026-08-06 10:54:57.621 UTC [10253] FATAL: database "slave" does not exist
127.30.218.20:59275 - accepting connections
I20260806 10:54:57.622792 31592 mini_ranger.cc:161] Starting Ranger...
I20260806 10:54:57.665752 31592 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 10:54:58.155691 31592 mini_ranger.cc:87] Created ranger Postgres database
I20260806 10:54:58.155869 31592 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ranger-admin
2026-08-06 10:54:58,499 [I] DB FLAVOR :POSTGRES
2026-08-06 10:54:58,500 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 10:54:58,500 [I] Checking connection..
2026-08-06 10:54:58,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:59275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 10:54:58,773 [I] Checking connection passed.
2026-08-06 10:54:58,773 [I] --------- Verifying version history table ---------
2026-08-06 10:54:58,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:59275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 10:54:59,016 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 10:54:59,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:59275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 10:54:59,285 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 10:54:59,285 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 10:54:59,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:59275/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 10:54:59.544 UTC [10317] WARNING: there is no transaction in progress
2026-08-06 10:54:59,551 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 10:54:59,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:59275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 10:54:59,801 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 10:54:59,802 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 10:54:59,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:59275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 10:55:00,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:59275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 10:55:00,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:59275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-49h3.c.gcp-upstream.internal','N') ;"
2026-08-06 10:55:00,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:59275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 10:55:00,824 [I] Table x_portal_user does not exist in database ranger
2026-08-06 10:55:00,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:59275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 10:55:01,081 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 10:55:01,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:59275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 10:55:01,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:59275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 10:55:01,586 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 10:55:01,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:59275/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 10:55:02.592 UTC [10426] WARNING: there is no transaction in progress
2026-08-06 10:55:02.609 UTC [10426] WARNING: there is no transaction in progress
2026-08-06 10:55:02.626 UTC [10426] WARNING: there is no transaction in progress
2026-08-06 10:55:02.642 UTC [10426] WARNING: there is no transaction in progress
2026-08-06 10:55:02.658 UTC [10426] WARNING: there is no transaction in progress
2026-08-06 10:55:02.675 UTC [10426] WARNING: there is no transaction in progress
2026-08-06 10:55:02.734 UTC [10426] WARNING: there is no transaction in progress
2026-08-06 10:55:02.741 UTC [10426] WARNING: there is no transaction in progress
2026-08-06 10:55:02.756 UTC [10426] WARNING: there is no transaction in progress
2026-08-06 10:55:02.767 UTC [10426] WARNING: there is no transaction in progress
2026-08-06 10:55:02.779 UTC [10426] WARNING: there is no transaction in progress
2026-08-06 10:55:02.794 UTC [10426] WARNING: there is no transaction in progress
2026-08-06 10:55:02.805 UTC [10426] WARNING: there is no transaction in progress
2026-08-06 10:55:02.812 UTC [10426] WARNING: there is no transaction in progress
2026-08-06 10:55:02.820 UTC [10426] WARNING: there is no transaction in progress
2026-08-06 10:55:02.828 UTC [10426] WARNING: there is no transaction in progress
2026-08-06 10:55:03,485 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 10:55:03,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:59275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 10:55:03,743 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 10:55:03,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:59275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 10:55:04,020 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 10:55:04,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:59275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 10:55:04,297 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 10:55:04,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:59275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 10:55:04,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:59275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 10:55:04,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:59275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-49h3.c.gcp-upstream.internal';"
2026-08-06 10:55:05,065 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 10:55:05,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:59275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 10:55:05,338 [I] DB_PATCHES have already been applied
I20260806 10:55:05.344903 31592 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ranger-admin:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskT4s7YX/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 10:55:05.345033 31592 mini_ranger.cc:193] Using host: 127.30.218.20
I20260806 10:55:05.368404 31592 mini_ranger.cc:227] Ranger admin URL: http://127.30.218.20:42739
Aug 06, 2026 10:55:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 10:55:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 10:55:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 10:55:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 10:55:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 10:55:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 10:55:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 10:55:06.446653 31592 mini_ranger.cc:160] Time spent starting Ranger: real 8.824s user 0.003s sys 0.084s
Aug 06, 2026 10:55:27 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 10:55:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 06, 2026 10:55:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 10:55:27 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 10:55:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 10:55:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 10:55:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 10:55:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 10:55:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 10:55:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 10:55:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 10:55:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 10:55:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 10:55:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 10:55:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 10:55:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 10:55:27 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 10:55:28 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 10:55:28.990868 31592 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.30.218.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.30.218.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.30.218.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/rangerkms_127.30.218.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.30.218.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/rangerkms_127.30.218.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.218.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.30.218.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.218.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/HTTP_127.30.218.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.218.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/HTTP_127.30.218.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 10:55:29 dist-test-slave-49h3 krb5kdc[10211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013729, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 10:55:29.213737 31592 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 10:55:29.244292 31592 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 10:55:30.061658 31592 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 10:55:30.061803 31592 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ranger-kms
I20260806 10:55:30.061933 31592 mini_ranger_kms.cc:227] Using postgres at 127.30.218.20:59275
2026-08-06 10:55:30,567 [I] DB FLAVOR :POSTGRES
2026-08-06 10:55:30,567 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 10:55:30,567 [I] Checking connection
2026-08-06 10:55:30,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:59275/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 10:55:30,818 [I] connection success
2026-08-06 10:55:30,818 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 10:55:30,819 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 10:55:30,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:59275/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 10:55:31,066 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 10:55:31,066 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 10:55:31,067 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 10:55:31,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:59275/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 10:55:31,362 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 10:55:31.847595 31592 mini_ranger_kms.cc:310] Created kms service
I20260806 10:55:32.053666 31592 mini_ranger_kms.cc:326] Created kudu user
I20260806 10:55:32.097169 31592 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 10:55:32 dist-test-slave-49h3 krb5kdc[10211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013729, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.30.218.20@KRBTEST.COM
I20260806 10:55:32.557845 31592 mini_ranger_kms.cc:384] Added ranger policy
I20260806 10:55:32.559337 31592 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ranger-kms:/tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 10:55:32.559417 31592 mini_ranger_kms.cc:240] Using host: 127.30.218.20
I20260806 10:55:32.589711 31592 mini_ranger_kms.cc:276] Ranger KMS PID: 10610
I20260806 10:55:32.589941 31592 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.30.218.20:40877
Aug 06, 2026 10:55:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 10:55:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 10:55:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 10:55:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 10:55:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 10:55:34 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260806 10:55:34.770722 31592 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.557s user 0.141s sys 0.418s
I20260806 10:55:34.771118 31592 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 10:55:34.771394 31592 mini_ranger_kms.cc:401] 127.30.218.20:40877/kms/v1/keys
Aug 06 10:55:34 dist-test-slave-49h3 krb5kdc[10211](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786013734, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.218.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 06 10:55:35 dist-test-slave-49h3 krb5kdc[10211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013734, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.218.20@KRBTEST.COM for HTTP/127.30.218.20@KRBTEST.COM
Aug 06 10:55:35 dist-test-slave-49h3 krb5kdc[10211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013734, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.218.20@KRBTEST.COM for HTTP/127.30.218.20@KRBTEST.COM
Aug 06 10:55:38 dist-test-slave-49h3 krb5kdc[10211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:55:38.275282 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:37777
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:42029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:37777
--ranger_config_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.218.20:40877/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:55:38.647529 10660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:55:38.647873 10660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:55:38.647993 10660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:55:38.655936 10660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:55:38.656087 10660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:55:38.656140 10660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:55:38.656225 10660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:55:38.656291 10660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:55:38.667395 10660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:42029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/master-0/wal
--ranger_kms_url=127.30.218.20:40877/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:37777
--ranger_config_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:37777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:55:38.668818 10660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:55:38.670298 10660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:55:38.685127 10666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:55:38.685738 10668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:55:38.685874 10665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:55:38.686870 10660 server_base.cc:1034] running on GCE node
I20260806 10:55:38.687777 10660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:55:38.689424 10660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:55:38.690699 10660 hybrid_clock.cc:648] HybridClock initialized: now 1786013738690635 us; error 64 us; skew 500 ppm
I20260806 10:55:38.691083 10660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:55:38 dist-test-slave-49h3 krb5kdc[10211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013738, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:55:38.699100 10660 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:55:38.700882 10660 webserver.cc:460] Webserver started at http://127.30.218.62:38049/ using document root <none> and password file <none>
I20260806 10:55:38.701654 10660 fs_manager.cc:359] Metadata directory not provided
I20260806 10:55:38.701769 10660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:55:38.702096 10660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 10:55:38 dist-test-slave-49h3 krb5kdc[10211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013738, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for HTTP/127.30.218.20@KRBTEST.COM
I20260806 10:55:38.762537 10660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "617cec634a214c64ad80946a65ff521c"
format_stamp: "Formatted at 2026-08-06 10:55:38 on dist-test-slave-49h3"
server_key: "8de03ec63cad0289a8d856ebd4fceb8b"
server_key_iv: "68a79c35d59e8816b4d5e7c3c1f43adc"
server_key_version: "kuduclusterkey@0"
I20260806 10:55:38.763504 10660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "617cec634a214c64ad80946a65ff521c"
format_stamp: "Formatted at 2026-08-06 10:55:38 on dist-test-slave-49h3"
server_key: "8de03ec63cad0289a8d856ebd4fceb8b"
server_key_iv: "68a79c35d59e8816b4d5e7c3c1f43adc"
server_key_version: "kuduclusterkey@0"
I20260806 10:55:38.769788 10660 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260806 10:55:38.791306 10675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:55:38.793380 10660 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 10:55:38.793687 10660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "617cec634a214c64ad80946a65ff521c"
format_stamp: "Formatted at 2026-08-06 10:55:38 on dist-test-slave-49h3"
server_key: "8de03ec63cad0289a8d856ebd4fceb8b"
server_key_iv: "68a79c35d59e8816b4d5e7c3c1f43adc"
server_key_version: "kuduclusterkey@0"
I20260806 10:55:38.793991 10660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:55:38.860433 10660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:55:38.865629 10660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:55:38.865978 10660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:55:38.886364 10660 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:37777
I20260806 10:55:38.886365 10727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:37777 every 8 connection(s)
I20260806 10:55:38.888243 10660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:55:38.892198 10728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:55:38.896749 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 10660
I20260806 10:55:38.896961 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:55:38.897814 31592 external_mini_cluster.cc:1428] Setting key a7ca14ec168728a382f27cc1fed6c1a1
I20260806 10:55:38.905584 10728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617cec634a214c64ad80946a65ff521c: Bootstrap starting.
I20260806 10:55:38.909801 10728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 617cec634a214c64ad80946a65ff521c: Neither blocks nor log segments found. Creating new log.
I20260806 10:55:38.910923 10728 log.cc:824] T 00000000000000000000000000000000 P 617cec634a214c64ad80946a65ff521c: Log is configured to *not* fsync() on all Append() calls
I20260806 10:55:38.914788 10728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617cec634a214c64ad80946a65ff521c: No bootstrap required, opened a new log
Aug 06 10:55:38 dist-test-slave-49h3 krb5kdc[10211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:55:38.924222 10731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:38.900154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54980 (local address 127.30.218.62:37777)
0806 10:55:38.901044 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0806 10:55:38.901063 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 10:55:38.901724 (+ 661us) server_negotiation.cc:407] Connection header received
0806 10:55:38.904296 (+ 2572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:55:38.904329 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:55:38.904941 (+ 612us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:55:38.905486 (+ 545us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:55:38.907172 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:38.909237 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:38.911189 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:38.912101 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:38.912304 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:55:38.916240 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:55:38.916249 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:55:38.916267 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:55:38.916309 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:55:38.919525 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:55:38.920393 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:55:38.920401 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:55:38.920409 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:55:38.920594 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:55:38.921002 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:55:38.921006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:55:38.921008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:55:38.921716 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:55:38.922046 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0806 10:55:38.922536 (+ 490us) server_negotiation.cc:299] Negotiation successful
0806 10:55:38.923031 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":413,"threads_started":1}
I20260806 10:55:38.927138 10728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 617cec634a214c64ad80946a65ff521c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617cec634a214c64ad80946a65ff521c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 37777 } }
I20260806 10:55:38.927747 10728 raft_consensus.cc:374] T 00000000000000000000000000000000 P 617cec634a214c64ad80946a65ff521c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:55:38.927868 10728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 617cec634a214c64ad80946a65ff521c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 617cec634a214c64ad80946a65ff521c, State: Initialized, Role: FOLLOWER
I20260806 10:55:38.928611 10728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 617cec634a214c64ad80946a65ff521c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617cec634a214c64ad80946a65ff521c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 37777 } }
I20260806 10:55:38.928911 10728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 617cec634a214c64ad80946a65ff521c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:55:38.929046 10728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 617cec634a214c64ad80946a65ff521c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:55:38.929219 10728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 617cec634a214c64ad80946a65ff521c [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:55:38.932868 10728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 617cec634a214c64ad80946a65ff521c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617cec634a214c64ad80946a65ff521c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 37777 } }
I20260806 10:55:38.933854 10728 leader_election.cc:302] T 00000000000000000000000000000000 P 617cec634a214c64ad80946a65ff521c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 617cec634a214c64ad80946a65ff521c; no voters:
I20260806 10:55:38.934619 10728 leader_election.cc:288] T 00000000000000000000000000000000 P 617cec634a214c64ad80946a65ff521c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:55:38.934715 10733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 617cec634a214c64ad80946a65ff521c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:55:38.935393 10733 raft_consensus.cc:686] T 00000000000000000000000000000000 P 617cec634a214c64ad80946a65ff521c [term 1 LEADER]: Becoming Leader. State: Replica: 617cec634a214c64ad80946a65ff521c, State: Running, Role: LEADER
I20260806 10:55:38.936187 10733 consensus_queue.cc:237] T 00000000000000000000000000000000 P 617cec634a214c64ad80946a65ff521c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617cec634a214c64ad80946a65ff521c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 37777 } }
I20260806 10:55:38.937903 10728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 617cec634a214c64ad80946a65ff521c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:55:38.939036 10734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617cec634a214c64ad80946a65ff521c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 617cec634a214c64ad80946a65ff521c. Latest consensus state: current_term: 1 leader_uuid: "617cec634a214c64ad80946a65ff521c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617cec634a214c64ad80946a65ff521c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 37777 } } }
I20260806 10:55:38.939455 10734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617cec634a214c64ad80946a65ff521c [sys.catalog]: This master's current role is: LEADER
I20260806 10:55:38.944335 10735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617cec634a214c64ad80946a65ff521c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "617cec634a214c64ad80946a65ff521c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617cec634a214c64ad80946a65ff521c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 37777 } } }
I20260806 10:55:38.944758 10735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617cec634a214c64ad80946a65ff521c [sys.catalog]: This master's current role is: LEADER
I20260806 10:55:38.948871 10738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:55:38.956610 10738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:55:38.984545 10738 catalog_manager.cc:1347] Generated new cluster ID: 3603c6526d3149b0a8cc63bec94ba650
I20260806 10:55:38.984701 10738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:55:39.013934 10738 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:55:39.016155 10738 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:55:39.027632 10738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 617cec634a214c64ad80946a65ff521c: Generated new TSK 0
I20260806 10:55:39.029081 10738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:55:39.032557 10728 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
WARNING: no policy specified for kudu/127.30.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
I20260806 10:55:39.982313 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.218.20:40877/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:37777
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:42029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 10:55:40 dist-test-slave-49h3 krb5kdc[10211](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786013740, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 10:55:40.436232 10773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:55:40.436792 10773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:55:40.437016 10773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:55:40.449324 10773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:55:40.449611 10773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:55:40.449854 10773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:55:40.466657 10773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:42029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.30.218.20:40877/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:37777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:55:40.468529 10773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:55:40.470544 10773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:55:40.488927 10783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:55:40.489267 10784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:55:40.489790 10786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:55:40.490392 10773 server_base.cc:1034] running on GCE node
I20260806 10:55:40.491192 10773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:55:40.492527 10773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:55:40.493789 10773 hybrid_clock.cc:648] HybridClock initialized: now 1786013740493674 us; error 85 us; skew 500 ppm
I20260806 10:55:40.494199 10773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:55:40 dist-test-slave-49h3 krb5kdc[10211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013740, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:55:40.498592 10773 init.cc:377] Logged in from keytab as kudu/127.30.218.1@KRBTEST.COM (short username kudu)
I20260806 10:55:40.500756 10773 webserver.cc:460] Webserver started at http://127.30.218.1:35193/ using document root <none> and password file <none>
I20260806 10:55:40.501780 10773 fs_manager.cc:359] Metadata directory not provided
I20260806 10:55:40.501940 10773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:55:40.502339 10773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 10:55:40 dist-test-slave-49h3 krb5kdc[10211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013740, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for HTTP/127.30.218.20@KRBTEST.COM
I20260806 10:55:40.522468 10773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "893ec918c35548f4a22dd3eb40599640"
format_stamp: "Formatted at 2026-08-06 10:55:40 on dist-test-slave-49h3"
server_key: "b63722e916c72e615361e7ce4c7b7c04"
server_key_iv: "eb233b0cf135a665bd6898215b6ca272"
server_key_version: "kuduclusterkey@0"
I20260806 10:55:40.523963 10773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "893ec918c35548f4a22dd3eb40599640"
format_stamp: "Formatted at 2026-08-06 10:55:40 on dist-test-slave-49h3"
server_key: "b63722e916c72e615361e7ce4c7b7c04"
server_key_iv: "eb233b0cf135a665bd6898215b6ca272"
server_key_version: "kuduclusterkey@0"
I20260806 10:55:40.548986 10773 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.008s sys 0.012s
I20260806 10:55:40.582347 10793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:55:40.584808 10773 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.005s sys 0.005s
I20260806 10:55:40.585152 10773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "893ec918c35548f4a22dd3eb40599640"
format_stamp: "Formatted at 2026-08-06 10:55:40 on dist-test-slave-49h3"
server_key: "b63722e916c72e615361e7ce4c7b7c04"
server_key_iv: "eb233b0cf135a665bd6898215b6ca272"
server_key_version: "kuduclusterkey@0"
I20260806 10:55:40.585578 10773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:55:40.633289 10773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:55:40.639475 10773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:55:40.639864 10773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:55:40.647421 10773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:55:40.650391 10773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:55:40.650518 10773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:55:40.650717 10773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:55:40.650787 10773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 10:55:40 dist-test-slave-49h3 krb5kdc[10211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013740, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for HTTP/127.30.218.20@KRBTEST.COM
I20260806 10:55:40.836910 10773 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:39145
I20260806 10:55:40.836927 10909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:39145 every 8 connection(s)
I20260806 10:55:40.838917 10773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:55:40.839443 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 10773
I20260806 10:55:40.839658 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:55:40.840447 31592 external_mini_cluster.cc:1428] Setting key 9c1d08c33ced044b794bcde46651562e
Aug 06 10:55:40 dist-test-slave-49h3 krb5kdc[10211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013740, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.218.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.2@KRBTEST.COM" created.
I20260806 10:55:40.887897 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:40.849841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:40771 (local address 127.30.218.62:37777)
0806 10:55:40.864304 (+ 14463us) server_negotiation.cc:206] Beginning negotiation
0806 10:55:40.864311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:55:40.864338 (+ 27us) server_negotiation.cc:407] Connection header received
0806 10:55:40.868526 (+ 4188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:55:40.868533 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:55:40.868669 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:55:40.868774 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:55:40.871219 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:40.872411 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:40.875010 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:40.875707 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:40.875810 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:55:40.880580 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:55:40.880589 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:55:40.880594 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:55:40.880639 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:55:40.883935 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:55:40.884827 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:55:40.884832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:55:40.884835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:55:40.884988 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:55:40.885562 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:55:40.885568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:55:40.885571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:55:40.886005 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:55:40.886179 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 10:55:40.887216 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0806 10:55:40.887475 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14319,"thread_start_us":14150,"threads_started":1}
I20260806 10:55:40.889267 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:40.850282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:37777 (local address 127.30.218.1:40771)
0806 10:55:40.862516 (+ 12234us) negotiation.cc:107] Waiting for socket to connect
0806 10:55:40.862609 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0806 10:55:40.867952 (+ 5343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:55:40.869291 (+ 1339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:55:40.869305 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:55:40.870053 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:55:40.870965 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:55:40.870992 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:55:40.872664 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:55:40.872670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:55:40.874814 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:55:40.874832 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:55:40.876201 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:55:40.876209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:55:40.876512 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:55:40.877379 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:55:40.877412 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:55:40.880267 (+ 2855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:55:40.884112 (+ 3845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:55:40.884123 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:55:40.884140 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:55:40.884678 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:55:40.885113 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:55:40.885119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:55:40.885122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:55:40.885412 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:55:40.886183 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:55:40.886192 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:55:40.886744 (+ 552us) client_negotiation.cc:769] Sending connection context
0806 10:55:40.887094 (+ 350us) client_negotiation.cc:240] Negotiation successful
0806 10:55:40.887721 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11832,"thread_start_us":11553,"threads_started":1}
I20260806 10:55:40.891429 10910 heartbeater.cc:347] Connected to a master server at 127.30.218.62:37777
I20260806 10:55:40.891839 10910 heartbeater.cc:464] Registering TS with master...
I20260806 10:55:40.892844 10910 heartbeater.cc:511] Master 127.30.218.62:37777 requested a full tablet report, sending...
I20260806 10:55:40.895562 10692 ts_manager.cc:194] Registered new tserver with Master: 893ec918c35548f4a22dd3eb40599640 (127.30.218.1:39145)
I20260806 10:55:40.897677 10692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.1@KRBTEST.COM'} at 127.30.218.1:40771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
I20260806 10:55:40.930122 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.218.20:40877/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:37777
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:42029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 10:55:41 dist-test-slave-49h3 krb5kdc[10211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013740, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for HTTP/127.30.218.20@KRBTEST.COM
W20260806 10:55:41.404021 10916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:55:41.404358 10916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:55:41.404529 10916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:55:41.416030 10916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:55:41.416317 10916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:55:41.416584 10916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:55:41.430528 10916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:42029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.30.218.20:40877/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:37777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:55:41.432096 10916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:55:41.433600 10916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:55:41.455341 10923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:55:41.455865 10924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:55:41.459553 10926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:55:41.459853 10916 server_base.cc:1034] running on GCE node
I20260806 10:55:41.460736 10916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:55:41.461902 10916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:55:41.463168 10916 hybrid_clock.cc:648] HybridClock initialized: now 1786013741463109 us; error 61 us; skew 500 ppm
I20260806 10:55:41.463521 10916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:55:41 dist-test-slave-49h3 krb5kdc[10211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013741, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:55:41.467236 10916 init.cc:377] Logged in from keytab as kudu/127.30.218.2@KRBTEST.COM (short username kudu)
I20260806 10:55:41.469228 10916 webserver.cc:460] Webserver started at http://127.30.218.2:35577/ using document root <none> and password file <none>
I20260806 10:55:41.470029 10916 fs_manager.cc:359] Metadata directory not provided
I20260806 10:55:41.470172 10916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:55:41.470499 10916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 10:55:41 dist-test-slave-49h3 krb5kdc[10211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013741, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for HTTP/127.30.218.20@KRBTEST.COM
I20260806 10:55:41.502135 10916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "75739086aef840fb8bb83ab9eff44a99"
format_stamp: "Formatted at 2026-08-06 10:55:41 on dist-test-slave-49h3"
server_key: "ac2d0ba5346bf6a3ef232323fcfaccf6"
server_key_iv: "5d1a3fe5f9f3da9dc7346db399fcd6f4"
server_key_version: "kuduclusterkey@0"
I20260806 10:55:41.503679 10916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "75739086aef840fb8bb83ab9eff44a99"
format_stamp: "Formatted at 2026-08-06 10:55:41 on dist-test-slave-49h3"
server_key: "ac2d0ba5346bf6a3ef232323fcfaccf6"
server_key_iv: "5d1a3fe5f9f3da9dc7346db399fcd6f4"
server_key_version: "kuduclusterkey@0"
I20260806 10:55:41.523720 10916 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.007s sys 0.011s
I20260806 10:55:41.552320 10940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:55:41.554523 10916 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.005s
I20260806 10:55:41.554772 10916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "75739086aef840fb8bb83ab9eff44a99"
format_stamp: "Formatted at 2026-08-06 10:55:41 on dist-test-slave-49h3"
server_key: "ac2d0ba5346bf6a3ef232323fcfaccf6"
server_key_iv: "5d1a3fe5f9f3da9dc7346db399fcd6f4"
server_key_version: "kuduclusterkey@0"
I20260806 10:55:41.555100 10916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:55:41.582909 10916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:55:41.589251 10916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:55:41.589702 10916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:55:41.591140 10916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:55:41.593571 10916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:55:41.593685 10916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:55:41.593822 10916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:55:41.593891 10916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:55:41.648126 10916 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:46527
I20260806 10:55:41.648155 11053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:46527 every 8 connection(s)
I20260806 10:55:41.650072 10916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:55:41.652149 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 10916
I20260806 10:55:41.652320 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:55:41.652933 31592 external_mini_cluster.cc:1428] Setting key 8607218f1e41dc89c5090909d6d0e6dc
Aug 06 10:55:41 dist-test-slave-49h3 krb5kdc[10211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013741, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:55:41.715008 11057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:41.668251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:37777 (local address 127.30.218.2:58541)
0806 10:55:41.679601 (+ 11350us) negotiation.cc:107] Waiting for socket to connect
0806 10:55:41.679693 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0806 10:55:41.686508 (+ 6815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:55:41.688272 (+ 1764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:55:41.688286 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:55:41.688981 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:55:41.689979 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:55:41.690017 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:55:41.692048 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:55:41.692054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:55:41.694299 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:55:41.694325 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:55:41.695529 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:55:41.695538 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:55:41.695859 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:55:41.696776 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:55:41.696811 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:55:41.699941 (+ 3130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:55:41.704449 (+ 4508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:55:41.704462 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:55:41.704485 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:55:41.705211 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:55:41.709826 (+ 4615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:55:41.709834 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:55:41.709839 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:55:41.710368 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:55:41.711480 (+ 1112us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:55:41.711491 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:55:41.712200 (+ 709us) client_negotiation.cc:769] Sending connection context
0806 10:55:41.712642 (+ 442us) client_negotiation.cc:240] Negotiation successful
0806 10:55:41.713342 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10850,"thread_start_us":13086,"threads_started":1}
I20260806 10:55:41.716406 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:41.667477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:58541 (local address 127.30.218.62:37777)
0806 10:55:41.686286 (+ 18809us) server_negotiation.cc:206] Beginning negotiation
0806 10:55:41.686294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:55:41.686332 (+ 38us) server_negotiation.cc:407] Connection header received
0806 10:55:41.687086 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:55:41.687093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:55:41.687254 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:55:41.687395 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:55:41.690287 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:41.691558 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:41.694531 (+ 2973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:41.695212 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:41.695368 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:55:41.700296 (+ 4928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:55:41.700303 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:55:41.700309 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:55:41.700362 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:55:41.704199 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:55:41.705419 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:55:41.705426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:55:41.705430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:55:41.705669 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:55:41.710617 (+ 4948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:55:41.710624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:55:41.710628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:55:41.711165 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:55:41.711370 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 10:55:41.715688 (+ 4318us) server_negotiation.cc:299] Negotiation successful
0806 10:55:41.716001 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18614,"thread_start_us":18955,"threads_started":1}
WARNING: no policy specified for kudu/127.30.218.3@KRBTEST.COM; defaulting to no policy
I20260806 10:55:41.719178 11054 heartbeater.cc:347] Connected to a master server at 127.30.218.62:37777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.3@KRBTEST.COM" created.
I20260806 10:55:41.719611 11054 heartbeater.cc:464] Registering TS with master...
I20260806 10:55:41.720752 11054 heartbeater.cc:511] Master 127.30.218.62:37777 requested a full tablet report, sending...
I20260806 10:55:41.722925 10692 ts_manager.cc:194] Registered new tserver with Master: 75739086aef840fb8bb83ab9eff44a99 (127.30.218.2:46527)
I20260806 10:55:41.724128 10692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.2@KRBTEST.COM'} at 127.30.218.2:58541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
W20260806 10:55:41.764804 31592 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.30.218.3: real 0.108s user 0.001s sys 0.029s
I20260806 10:55:41.765919 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.218.20:40877/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:37777
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:42029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:55:41.902355 10910 heartbeater.cc:503] Master 127.30.218.62:37777 was elected leader, sending a full tablet report...
W20260806 10:55:42.194624 11061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:55:42.194929 11061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:55:42.195051 11061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:55:42.203226 11061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:55:42.203374 11061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:55:42.203540 11061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:55:42.214293 11061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:42029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.30.218.20:40877/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:37777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:55:42.215754 11061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:55:42.217206 11061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:55:42.231151 11066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:55:42.231184 11069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:55:42.231112 11067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:55:42.231977 11061 server_base.cc:1034] running on GCE node
I20260806 10:55:42.232679 11061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:55:42.233789 11061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:55:42.235009 11061 hybrid_clock.cc:648] HybridClock initialized: now 1786013742234958 us; error 50 us; skew 500 ppm
I20260806 10:55:42.235365 11061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:55:42 dist-test-slave-49h3 krb5kdc[10211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013742, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:55:42.238847 11061 init.cc:377] Logged in from keytab as kudu/127.30.218.3@KRBTEST.COM (short username kudu)
I20260806 10:55:42.240635 11061 webserver.cc:460] Webserver started at http://127.30.218.3:38965/ using document root <none> and password file <none>
I20260806 10:55:42.241405 11061 fs_manager.cc:359] Metadata directory not provided
I20260806 10:55:42.241581 11061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:55:42.241925 11061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 10:55:42 dist-test-slave-49h3 krb5kdc[10211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013742, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for HTTP/127.30.218.20@KRBTEST.COM
I20260806 10:55:42.261111 11061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "b9d43c880cfe4373bf15e9e00a59bd2f"
format_stamp: "Formatted at 2026-08-06 10:55:42 on dist-test-slave-49h3"
server_key: "2af24b64499a3e367ba702accaf646a8"
server_key_iv: "d768686d24a6200352f42cdcd6169f45"
server_key_version: "kuduclusterkey@0"
I20260806 10:55:42.262347 11061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "b9d43c880cfe4373bf15e9e00a59bd2f"
format_stamp: "Formatted at 2026-08-06 10:55:42 on dist-test-slave-49h3"
server_key: "2af24b64499a3e367ba702accaf646a8"
server_key_iv: "d768686d24a6200352f42cdcd6169f45"
server_key_version: "kuduclusterkey@0"
I20260806 10:55:42.270309 11061 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260806 10:55:42.286887 11076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:55:42.288621 11061 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 10:55:42.288815 11061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "b9d43c880cfe4373bf15e9e00a59bd2f"
format_stamp: "Formatted at 2026-08-06 10:55:42 on dist-test-slave-49h3"
server_key: "2af24b64499a3e367ba702accaf646a8"
server_key_iv: "d768686d24a6200352f42cdcd6169f45"
server_key_version: "kuduclusterkey@0"
I20260806 10:55:42.289089 11061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:55:42.328347 11061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:55:42.332351 11061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:55:42.332625 11061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:55:42.333664 11061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:55:42.335130 11061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:55:42.335204 11061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:55:42.335297 11061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:55:42.335343 11061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:55:42.367458 11061 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:35783
I20260806 10:55:42.367480 11189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:35783 every 8 connection(s)
I20260806 10:55:42.368894 11061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:55:42.375669 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 11061
I20260806 10:55:42.375867 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:55:42.376437 31592 external_mini_cluster.cc:1428] Setting key 00d8614e63b0141c518d2886e0dc6c82
Aug 06 10:55:42 dist-test-slave-49h3 krb5kdc[10211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013742, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:55:42.391610 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:42.372040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:57287 (local address 127.30.218.62:37777)
0806 10:55:42.372561 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0806 10:55:42.372566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:55:42.373488 (+ 922us) server_negotiation.cc:407] Connection header received
0806 10:55:42.375337 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:55:42.375343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:55:42.375455 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:55:42.375558 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:55:42.377550 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.378538 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.380417 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.380865 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.380966 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:55:42.384804 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:55:42.384810 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:55:42.384814 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:55:42.384867 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:55:42.387777 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:55:42.388777 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:55:42.388783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:55:42.388785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:55:42.388944 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:55:42.389400 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:55:42.389404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:55:42.389411 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:55:42.389798 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:55:42.389922 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 10:55:42.390961 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0806 10:55:42.391217 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":273,"threads_started":1}
I20260806 10:55:42.392907 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:42.372536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:37777 (local address 127.30.218.3:57287)
0806 10:55:42.373289 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0806 10:55:42.373334 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 10:55:42.375026 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:55:42.375874 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:55:42.375887 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:55:42.376510 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:55:42.377330 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:55:42.377357 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.378723 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.378728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:55:42.380264 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:55:42.380278 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.381070 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.381075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:55:42.381302 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:55:42.382013 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:55:42.382038 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:55:42.384470 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:55:42.387990 (+ 3520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:55:42.387999 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:55:42.388016 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:55:42.388602 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:55:42.389056 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:55:42.389059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:55:42.389062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:55:42.389283 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:55:42.389946 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:55:42.389954 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:55:42.390496 (+ 542us) client_negotiation.cc:769] Sending connection context
0806 10:55:42.390821 (+ 325us) client_negotiation.cc:240] Negotiation successful
0806 10:55:42.391381 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":286,"threads_started":1}
I20260806 10:55:42.394539 11190 heartbeater.cc:347] Connected to a master server at 127.30.218.62:37777
I20260806 10:55:42.394892 11190 heartbeater.cc:464] Registering TS with master...
I20260806 10:55:42.395752 11190 heartbeater.cc:511] Master 127.30.218.62:37777 requested a full tablet report, sending...
I20260806 10:55:42.397420 10692 ts_manager.cc:194] Registered new tserver with Master: b9d43c880cfe4373bf15e9e00a59bd2f (127.30.218.3:35783)
I20260806 10:55:42.398490 10692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.3@KRBTEST.COM'} at 127.30.218.3:57287
I20260806 10:55:42.406059 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:55:42.424139 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:42.411073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54986 (local address 127.30.218.62:37777)
0806 10:55:42.411320 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 10:55:42.411327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:55:42.411688 (+ 361us) server_negotiation.cc:407] Connection header received
0806 10:55:42.411864 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:55:42.411870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:55:42.411955 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:55:42.412034 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:55:42.413617 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.414371 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.415858 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.416328 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.416430 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:55:42.418949 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:55:42.418955 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:55:42.418958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:55:42.418998 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:55:42.421762 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:55:42.422449 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:55:42.422454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:55:42.422456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:55:42.422603 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:55:42.422898 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:55:42.422901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:55:42.422903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:55:42.423187 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:55:42.423394 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 10:55:42.423636 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 10:55:42.423837 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260806 10:55:42.428660 10692 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:55:42.436220 10692 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:55:42.476199 11205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:42.450590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:39145 (local address 127.0.0.1:46734)
0806 10:55:42.463904 (+ 13314us) negotiation.cc:107] Waiting for socket to connect
0806 10:55:42.463950 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 10:55:42.464332 (+ 382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:55:42.467772 (+ 3440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:55:42.467778 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:55:42.467813 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:55:42.468300 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:55:42.468316 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.470291 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.470300 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:55:42.473018 (+ 2718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:55:42.473036 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.475033 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.475040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:55:42.475248 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:55:42.475327 (+ 79us) client_negotiation.cc:769] Sending connection context
0806 10:55:42.475490 (+ 163us) client_negotiation.cc:240] Negotiation successful
0806 10:55:42.475734 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13006,"mutex_wait_us":99,"thread_start_us":14200,"threads_started":1}
I20260806 10:55:42.476321 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:42.451282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33394 (local address 127.30.218.2:46527)
0806 10:55:42.463286 (+ 12004us) server_negotiation.cc:206] Beginning negotiation
0806 10:55:42.463299 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 10:55:42.464654 (+ 1355us) server_negotiation.cc:407] Connection header received
0806 10:55:42.464833 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:55:42.464840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:55:42.465067 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:55:42.465253 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:55:42.465962 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.469956 (+ 3994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.473283 (+ 3327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.474783 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.474936 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:55:42.475147 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 10:55:42.475779 (+ 632us) server_negotiation.cc:299] Negotiation successful
0806 10:55:42.475972 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11741,"thread_start_us":11564,"threads_started":1}
I20260806 10:55:42.476497 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:42.450830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46734 (local address 127.30.218.1:39145)
0806 10:55:42.467063 (+ 16233us) server_negotiation.cc:206] Beginning negotiation
0806 10:55:42.467073 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:55:42.467110 (+ 37us) server_negotiation.cc:407] Connection header received
0806 10:55:42.467226 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:55:42.467235 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:55:42.467541 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:55:42.467774 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:55:42.468536 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.470078 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.473202 (+ 3124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.474784 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.474936 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:55:42.475183 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0806 10:55:42.475978 (+ 795us) server_negotiation.cc:299] Negotiation successful
0806 10:55:42.476155 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15992,"thread_start_us":17187,"threads_started":1}
I20260806 10:55:42.476565 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:42.466100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:35783 (local address 127.0.0.1:41524)
0806 10:55:42.468721 (+ 2621us) negotiation.cc:107] Waiting for socket to connect
0806 10:55:42.468749 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 10:55:42.468881 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:55:42.470076 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:55:42.470080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:55:42.470107 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:55:42.470559 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:55:42.470569 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.472520 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.472528 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:55:42.474173 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:55:42.474185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.475769 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.475774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:55:42.475944 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:55:42.475978 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 10:55:42.476042 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 10:55:42.476176 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2472,"thread_start_us":2294,"threads_started":1}
I20260806 10:55:42.476966 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:42.451072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:46527 (local address 127.0.0.1:33394)
0806 10:55:42.464485 (+ 13413us) negotiation.cc:107] Waiting for socket to connect
0806 10:55:42.464508 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:55:42.464700 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:55:42.465286 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:55:42.465294 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:55:42.465325 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:55:42.465771 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:55:42.465785 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.470121 (+ 4336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.470128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:55:42.473109 (+ 2981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:55:42.473124 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.475317 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.475322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:55:42.475441 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:55:42.475471 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 10:55:42.475560 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 10:55:42.476569 (+ 1009us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13258,"mutex_wait_us":903,"thread_start_us":12331,"threads_started":1}
I20260806 10:55:42.477802 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:42.465832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41524 (local address 127.30.218.3:35783)
0806 10:55:42.469550 (+ 3718us) server_negotiation.cc:206] Beginning negotiation
0806 10:55:42.469560 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:55:42.469600 (+ 40us) server_negotiation.cc:407] Connection header received
0806 10:55:42.469698 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:55:42.469706 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:55:42.469895 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:55:42.470075 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:55:42.470749 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.472327 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.474332 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.475538 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.476653 (+ 1115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:55:42.477181 (+ 528us) server_negotiation.cc:1035] Waiting for connection context
0806 10:55:42.477269 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 10:55:42.477429 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3509,"thread_start_us":3317,"threads_started":1}
I20260806 10:55:42.481122 10844 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6505a132e44610909d3eed7c91c177 (DEFAULT_TABLE table=test-table [id=f2daa55abf62404dbc57a82a5c6d4d30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:55:42.481251 11124 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6505a132e44610909d3eed7c91c177 (DEFAULT_TABLE table=test-table [id=f2daa55abf62404dbc57a82a5c6d4d30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:55:42.481251 10989 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6505a132e44610909d3eed7c91c177 (DEFAULT_TABLE table=test-table [id=f2daa55abf62404dbc57a82a5c6d4d30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:55:42.482707 10844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6505a132e44610909d3eed7c91c177. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:55:42.482730 11124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6505a132e44610909d3eed7c91c177. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:55:42.482730 10989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6505a132e44610909d3eed7c91c177. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:55:42.496009 11209 tablet_bootstrap.cc:492] T 3f6505a132e44610909d3eed7c91c177 P 893ec918c35548f4a22dd3eb40599640: Bootstrap starting.
I20260806 10:55:42.498817 11211 tablet_bootstrap.cc:492] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f: Bootstrap starting.
I20260806 10:55:42.499488 11210 tablet_bootstrap.cc:492] T 3f6505a132e44610909d3eed7c91c177 P 75739086aef840fb8bb83ab9eff44a99: Bootstrap starting.
I20260806 10:55:42.500864 11209 tablet_bootstrap.cc:654] T 3f6505a132e44610909d3eed7c91c177 P 893ec918c35548f4a22dd3eb40599640: Neither blocks nor log segments found. Creating new log.
I20260806 10:55:42.502731 11211 tablet_bootstrap.cc:654] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f: Neither blocks nor log segments found. Creating new log.
I20260806 10:55:42.502859 11210 tablet_bootstrap.cc:654] T 3f6505a132e44610909d3eed7c91c177 P 75739086aef840fb8bb83ab9eff44a99: Neither blocks nor log segments found. Creating new log.
I20260806 10:55:42.503446 11209 log.cc:824] T 3f6505a132e44610909d3eed7c91c177 P 893ec918c35548f4a22dd3eb40599640: Log is configured to *not* fsync() on all Append() calls
I20260806 10:55:42.504086 11210 log.cc:824] T 3f6505a132e44610909d3eed7c91c177 P 75739086aef840fb8bb83ab9eff44a99: Log is configured to *not* fsync() on all Append() calls
I20260806 10:55:42.504108 11211 log.cc:824] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f: Log is configured to *not* fsync() on all Append() calls
I20260806 10:55:42.507030 11209 tablet_bootstrap.cc:492] T 3f6505a132e44610909d3eed7c91c177 P 893ec918c35548f4a22dd3eb40599640: No bootstrap required, opened a new log
I20260806 10:55:42.507326 11209 ts_tablet_manager.cc:1397] T 3f6505a132e44610909d3eed7c91c177 P 893ec918c35548f4a22dd3eb40599640: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260806 10:55:42.507510 11210 tablet_bootstrap.cc:492] T 3f6505a132e44610909d3eed7c91c177 P 75739086aef840fb8bb83ab9eff44a99: No bootstrap required, opened a new log
I20260806 10:55:42.507776 11211 tablet_bootstrap.cc:492] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f: No bootstrap required, opened a new log
I20260806 10:55:42.507803 11210 ts_tablet_manager.cc:1397] T 3f6505a132e44610909d3eed7c91c177 P 75739086aef840fb8bb83ab9eff44a99: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260806 10:55:42.508085 11211 ts_tablet_manager.cc:1397] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260806 10:55:42.515877 11209 raft_consensus.cc:348] T 3f6505a132e44610909d3eed7c91c177 P 893ec918c35548f4a22dd3eb40599640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893ec918c35548f4a22dd3eb40599640" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 39145 } } peers { permanent_uuid: "b9d43c880cfe4373bf15e9e00a59bd2f" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35783 } } peers { permanent_uuid: "75739086aef840fb8bb83ab9eff44a99" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 46527 } }
I20260806 10:55:42.516366 11209 raft_consensus.cc:374] T 3f6505a132e44610909d3eed7c91c177 P 893ec918c35548f4a22dd3eb40599640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:55:42.516497 11209 raft_consensus.cc:729] T 3f6505a132e44610909d3eed7c91c177 P 893ec918c35548f4a22dd3eb40599640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893ec918c35548f4a22dd3eb40599640, State: Initialized, Role: FOLLOWER
I20260806 10:55:42.517169 11209 consensus_queue.cc:260] T 3f6505a132e44610909d3eed7c91c177 P 893ec918c35548f4a22dd3eb40599640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893ec918c35548f4a22dd3eb40599640" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 39145 } } peers { permanent_uuid: "b9d43c880cfe4373bf15e9e00a59bd2f" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35783 } } peers { permanent_uuid: "75739086aef840fb8bb83ab9eff44a99" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 46527 } }
I20260806 10:55:42.519133 11209 ts_tablet_manager.cc:1428] T 3f6505a132e44610909d3eed7c91c177 P 893ec918c35548f4a22dd3eb40599640: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260806 10:55:42.519094 11211 raft_consensus.cc:348] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893ec918c35548f4a22dd3eb40599640" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 39145 } } peers { permanent_uuid: "b9d43c880cfe4373bf15e9e00a59bd2f" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35783 } } peers { permanent_uuid: "75739086aef840fb8bb83ab9eff44a99" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 46527 } }
I20260806 10:55:42.519704 11211 raft_consensus.cc:374] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:55:42.519860 11211 raft_consensus.cc:729] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d43c880cfe4373bf15e9e00a59bd2f, State: Initialized, Role: FOLLOWER
I20260806 10:55:42.519873 11210 raft_consensus.cc:348] T 3f6505a132e44610909d3eed7c91c177 P 75739086aef840fb8bb83ab9eff44a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893ec918c35548f4a22dd3eb40599640" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 39145 } } peers { permanent_uuid: "b9d43c880cfe4373bf15e9e00a59bd2f" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35783 } } peers { permanent_uuid: "75739086aef840fb8bb83ab9eff44a99" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 46527 } }
I20260806 10:55:42.520340 11210 raft_consensus.cc:374] T 3f6505a132e44610909d3eed7c91c177 P 75739086aef840fb8bb83ab9eff44a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:55:42.520454 11210 raft_consensus.cc:729] T 3f6505a132e44610909d3eed7c91c177 P 75739086aef840fb8bb83ab9eff44a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75739086aef840fb8bb83ab9eff44a99, State: Initialized, Role: FOLLOWER
I20260806 10:55:42.520529 11211 consensus_queue.cc:260] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893ec918c35548f4a22dd3eb40599640" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 39145 } } peers { permanent_uuid: "b9d43c880cfe4373bf15e9e00a59bd2f" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35783 } } peers { permanent_uuid: "75739086aef840fb8bb83ab9eff44a99" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 46527 } }
I20260806 10:55:42.521087 11210 consensus_queue.cc:260] T 3f6505a132e44610909d3eed7c91c177 P 75739086aef840fb8bb83ab9eff44a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893ec918c35548f4a22dd3eb40599640" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 39145 } } peers { permanent_uuid: "b9d43c880cfe4373bf15e9e00a59bd2f" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35783 } } peers { permanent_uuid: "75739086aef840fb8bb83ab9eff44a99" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 46527 } }
I20260806 10:55:42.522423 11211 ts_tablet_manager.cc:1428] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260806 10:55:42.523468 11190 heartbeater.cc:503] Master 127.30.218.62:37777 was elected leader, sending a full tablet report...
I20260806 10:55:42.524214 11210 ts_tablet_manager.cc:1428] T 3f6505a132e44610909d3eed7c91c177 P 75739086aef840fb8bb83ab9eff44a99: Time spent starting tablet: real 0.016s user 0.007s sys 0.007s
I20260806 10:55:42.524703 11054 heartbeater.cc:503] Master 127.30.218.62:37777 was elected leader, sending a full tablet report...
W20260806 10:55:42.618736 10914 tablet.cc:2369] T 3f6505a132e44610909d3eed7c91c177 P 893ec918c35548f4a22dd3eb40599640: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:55:42.620723 11191 tablet.cc:2369] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:55:42.687249 11216 raft_consensus.cc:482] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:55:42.687561 11216 raft_consensus.cc:504] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893ec918c35548f4a22dd3eb40599640" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 39145 } } peers { permanent_uuid: "b9d43c880cfe4373bf15e9e00a59bd2f" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35783 } } peers { permanent_uuid: "75739086aef840fb8bb83ab9eff44a99" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 46527 } }
I20260806 10:55:42.689479 11216 leader_election.cc:288] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 893ec918c35548f4a22dd3eb40599640 (127.30.218.1:39145), 75739086aef840fb8bb83ab9eff44a99 (127.30.218.2:46527)
I20260806 10:55:42.695269 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:42.689787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:46527 (local address 127.30.218.3:57347)
0806 10:55:42.690001 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 10:55:42.690023 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:55:42.690167 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:55:42.690577 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:55:42.690582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:55:42.690615 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:55:42.690789 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:55:42.690797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.691857 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.691862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:55:42.693316 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:55:42.693329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.694503 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.694509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:55:42.694700 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:55:42.694726 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 10:55:42.694811 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 10:55:42.694958 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":1}
I20260806 10:55:42.695253 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:42.689971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:57347 (local address 127.30.218.2:46527)
0806 10:55:42.690213 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 10:55:42.690218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:55:42.690254 (+ 36us) server_negotiation.cc:407] Connection header received
0806 10:55:42.690333 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:55:42.690339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:55:42.690416 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:55:42.690511 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:55:42.690948 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.691717 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.693503 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.694360 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.694508 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:55:42.694585 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 10:55:42.694858 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 10:55:42.694987 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260806 10:55:42.696310 11055 tablet.cc:2369] T 3f6505a132e44610909d3eed7c91c177 P 75739086aef840fb8bb83ab9eff44a99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:55:42.696465 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:42.689787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:39145 (local address 127.30.218.3:33247)
0806 10:55:42.690783 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0806 10:55:42.690799 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:55:42.690917 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:55:42.691225 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:55:42.691229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:55:42.691275 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:55:42.691630 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:55:42.691640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.692773 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.692777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:55:42.694286 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:55:42.694299 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.695660 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.695667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:55:42.695828 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:55:42.695860 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 10:55:42.695959 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 10:55:42.696091 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"mutex_wait_us":212,"thread_start_us":387,"threads_started":1}
I20260806 10:55:42.696456 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:42.690043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:33247 (local address 127.30.218.1:39145)
0806 10:55:42.690311 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0806 10:55:42.690317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:55:42.690891 (+ 574us) server_negotiation.cc:407] Connection header received
0806 10:55:42.691048 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:55:42.691052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:55:42.691113 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:55:42.691188 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:55:42.691780 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.692629 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.694448 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.695348 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.695515 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:55:42.695589 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 10:55:42.696036 (+ 447us) server_negotiation.cc:299] Negotiation successful
0806 10:55:42.696207 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260806 10:55:42.696601 11009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6505a132e44610909d3eed7c91c177" candidate_uuid: "b9d43c880cfe4373bf15e9e00a59bd2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75739086aef840fb8bb83ab9eff44a99" is_pre_election: true
I20260806 10:55:42.697281 11009 raft_consensus.cc:2454] T 3f6505a132e44610909d3eed7c91c177 P 75739086aef840fb8bb83ab9eff44a99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d43c880cfe4373bf15e9e00a59bd2f in term 0.
I20260806 10:55:42.697542 10865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6505a132e44610909d3eed7c91c177" candidate_uuid: "b9d43c880cfe4373bf15e9e00a59bd2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893ec918c35548f4a22dd3eb40599640" is_pre_election: true
I20260806 10:55:42.698063 10865 raft_consensus.cc:2454] T 3f6505a132e44610909d3eed7c91c177 P 893ec918c35548f4a22dd3eb40599640 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d43c880cfe4373bf15e9e00a59bd2f in term 0.
I20260806 10:55:42.698048 11080 leader_election.cc:302] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75739086aef840fb8bb83ab9eff44a99, b9d43c880cfe4373bf15e9e00a59bd2f; no voters:
I20260806 10:55:42.698436 11216 raft_consensus.cc:2781] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:55:42.698566 11216 raft_consensus.cc:482] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:55:42.698707 11216 raft_consensus.cc:3037] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:55:42.701072 11216 raft_consensus.cc:504] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893ec918c35548f4a22dd3eb40599640" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 39145 } } peers { permanent_uuid: "b9d43c880cfe4373bf15e9e00a59bd2f" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35783 } } peers { permanent_uuid: "75739086aef840fb8bb83ab9eff44a99" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 46527 } }
I20260806 10:55:42.701711 11216 leader_election.cc:288] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f [CANDIDATE]: Term 1 election: Requested vote from peers 893ec918c35548f4a22dd3eb40599640 (127.30.218.1:39145), 75739086aef840fb8bb83ab9eff44a99 (127.30.218.2:46527)
I20260806 10:55:42.702250 11009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6505a132e44610909d3eed7c91c177" candidate_uuid: "b9d43c880cfe4373bf15e9e00a59bd2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75739086aef840fb8bb83ab9eff44a99"
I20260806 10:55:42.702250 10865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6505a132e44610909d3eed7c91c177" candidate_uuid: "b9d43c880cfe4373bf15e9e00a59bd2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893ec918c35548f4a22dd3eb40599640"
I20260806 10:55:42.702468 11009 raft_consensus.cc:3037] T 3f6505a132e44610909d3eed7c91c177 P 75739086aef840fb8bb83ab9eff44a99 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:55:42.702468 10865 raft_consensus.cc:3037] T 3f6505a132e44610909d3eed7c91c177 P 893ec918c35548f4a22dd3eb40599640 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:55:42.705695 10865 raft_consensus.cc:2454] T 3f6505a132e44610909d3eed7c91c177 P 893ec918c35548f4a22dd3eb40599640 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d43c880cfe4373bf15e9e00a59bd2f in term 1.
I20260806 10:55:42.705695 11009 raft_consensus.cc:2454] T 3f6505a132e44610909d3eed7c91c177 P 75739086aef840fb8bb83ab9eff44a99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d43c880cfe4373bf15e9e00a59bd2f in term 1.
I20260806 10:55:42.706166 11079 leader_election.cc:302] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 893ec918c35548f4a22dd3eb40599640, b9d43c880cfe4373bf15e9e00a59bd2f; no voters:
I20260806 10:55:42.706485 11216 raft_consensus.cc:2781] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:55:42.706905 11216 raft_consensus.cc:686] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f [term 1 LEADER]: Becoming Leader. State: Replica: b9d43c880cfe4373bf15e9e00a59bd2f, State: Running, Role: LEADER
I20260806 10:55:42.707397 11216 consensus_queue.cc:237] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893ec918c35548f4a22dd3eb40599640" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 39145 } } peers { permanent_uuid: "b9d43c880cfe4373bf15e9e00a59bd2f" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35783 } } peers { permanent_uuid: "75739086aef840fb8bb83ab9eff44a99" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 46527 } }
I20260806 10:55:42.712002 10692 catalog_manager.cc:5495] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f reported cstate change: term changed from 0 to 1, leader changed from <none> to b9d43c880cfe4373bf15e9e00a59bd2f (127.30.218.3). New cstate: current_term: 1 leader_uuid: "b9d43c880cfe4373bf15e9e00a59bd2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893ec918c35548f4a22dd3eb40599640" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 39145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d43c880cfe4373bf15e9e00a59bd2f" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75739086aef840fb8bb83ab9eff44a99" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 46527 } health_report { overall_health: UNKNOWN } } }
I20260806 10:55:42.780424 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:42.775288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41528 (local address 127.30.218.3:35783)
0806 10:55:42.775480 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 10:55:42.775484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:55:42.775503 (+ 19us) server_negotiation.cc:407] Connection header received
0806 10:55:42.775615 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:55:42.775619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:55:42.775706 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:55:42.775835 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:55:42.776201 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.776936 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.778396 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.778805 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.778900 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:55:42.779411 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:55:42.779928 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:55:42.780033 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 10:55:42.780139 (+ 106us) server_negotiation.cc:299] Negotiation successful
0806 10:55:42.780214 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 10:55:42.789075 10865 raft_consensus.cc:1261] T 3f6505a132e44610909d3eed7c91c177 P 893ec918c35548f4a22dd3eb40599640 [term 1 FOLLOWER]: Refusing update from remote peer b9d43c880cfe4373bf15e9e00a59bd2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:55:42.789076 11009 raft_consensus.cc:1261] T 3f6505a132e44610909d3eed7c91c177 P 75739086aef840fb8bb83ab9eff44a99 [term 1 FOLLOWER]: Refusing update from remote peer b9d43c880cfe4373bf15e9e00a59bd2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:55:42.790234 11219 consensus_queue.cc:1035] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "893ec918c35548f4a22dd3eb40599640" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 39145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:55:42.790598 11216 consensus_queue.cc:1035] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "75739086aef840fb8bb83ab9eff44a99" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 46527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:55:42.802625 11222 mvcc.cc:205] Tried to move back new op lower bound from 7315512290453282816 to 7315512290133139456. Current Snapshot: MvccSnapshot[applied={T|T < 7315512290453282816}]
I20260806 10:55:42.802949 11224 mvcc.cc:205] Tried to move back new op lower bound from 7315512290453282816 to 7315512290133139456. Current Snapshot: MvccSnapshot[applied={T|T < 7315512290453282816}]
I20260806 10:55:42.818710 10691 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:55:42.819020 10691 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:55:42.823383 10691 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 617cec634a214c64ad80946a65ff521c: Sending DeleteTablet for 3 replicas of tablet 3f6505a132e44610909d3eed7c91c177
I20260806 10:55:42.824837 11124 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f6505a132e44610909d3eed7c91c177 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:55:42 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:41524
I20260806 10:55:42.824908 10844 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f6505a132e44610909d3eed7c91c177 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:55:42 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:46734
I20260806 10:55:42.824892 10989 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f6505a132e44610909d3eed7c91c177 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:55:42 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:33394
I20260806 10:55:42.826977 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 10773
I20260806 10:55:42.834069 11234 tablet_replica.cc:330] T 3f6505a132e44610909d3eed7c91c177 P 75739086aef840fb8bb83ab9eff44a99: stopping tablet replica
I20260806 10:55:42.834637 11234 raft_consensus.cc:2229] T 3f6505a132e44610909d3eed7c91c177 P 75739086aef840fb8bb83ab9eff44a99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:55:42.835096 11233 tablet_replica.cc:330] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f: stopping tablet replica
I20260806 10:55:42.835323 11234 raft_consensus.cc:2258] T 3f6505a132e44610909d3eed7c91c177 P 75739086aef840fb8bb83ab9eff44a99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:55:42.835634 11233 raft_consensus.cc:2229] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:55:42.836555 11233 raft_consensus.cc:2258] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:55:42.837414 11232 tablet_replica.cc:330] T 3f6505a132e44610909d3eed7c91c177 P 893ec918c35548f4a22dd3eb40599640: stopping tablet replica
I20260806 10:55:42.839675 11233 ts_tablet_manager.cc:1905] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:55:42.840319 11232 raft_consensus.cc:2229] T 3f6505a132e44610909d3eed7c91c177 P 893ec918c35548f4a22dd3eb40599640 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:55:42.846923 11234 ts_tablet_manager.cc:1905] T 3f6505a132e44610909d3eed7c91c177 P 75739086aef840fb8bb83ab9eff44a99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:55:42.847429 11232 raft_consensus.cc:2258] T 3f6505a132e44610909d3eed7c91c177 P 893ec918c35548f4a22dd3eb40599640 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 06 10:55:42 dist-test-slave-49h3 krb5kdc[10211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.1@KRBTEST.COM
I20260806 10:55:42.852125 11232 ts_tablet_manager.cc:1905] T 3f6505a132e44610909d3eed7c91c177 P 893ec918c35548f4a22dd3eb40599640: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:55:42.855024 11234 ts_tablet_manager.cc:1918] T 3f6505a132e44610909d3eed7c91c177 P 75739086aef840fb8bb83ab9eff44a99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:55:42.855352 11234 log.cc:1196] T 3f6505a132e44610909d3eed7c91c177 P 75739086aef840fb8bb83ab9eff44a99: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/3f6505a132e44610909d3eed7c91c177
I20260806 10:55:42.856007 11234 ts_tablet_manager.cc:1939] T 3f6505a132e44610909d3eed7c91c177 P 75739086aef840fb8bb83ab9eff44a99: Deleting consensus metadata
I20260806 10:55:42.858598 10679 catalog_manager.cc:4841] TS 75739086aef840fb8bb83ab9eff44a99 (127.30.218.2:46527): tablet 3f6505a132e44610909d3eed7c91c177 (table test-table [id=f2daa55abf62404dbc57a82a5c6d4d30]) successfully deleted
I20260806 10:55:42.860417 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:42.828022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46738 (local address 127.30.218.1:39145)
0806 10:55:42.828330 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0806 10:55:42.828337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:55:42.834158 (+ 5821us) server_negotiation.cc:407] Connection header received
0806 10:55:42.834372 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:55:42.834377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:55:42.834506 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:55:42.834636 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:55:42.839582 (+ 4946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.840901 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.843088 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:42.843846 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:42.843970 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:55:42.851373 (+ 7403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:55:42.851389 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:55:42.851407 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:55:42.851458 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:55:42.855886 (+ 4428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:55:42.857070 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:55:42.857081 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:55:42.857085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:55:42.857301 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:55:42.857904 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:55:42.857909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:55:42.857912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:55:42.858590 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:55:42.858922 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0806 10:55:42.859720 (+ 798us) server_negotiation.cc:299] Negotiation successful
0806 10:55:42.859995 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260806 10:55:42.862694 10905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:55:42.863059 11233 ts_tablet_manager.cc:1918] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:55:42.863469 11233 log.cc:1196] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/3f6505a132e44610909d3eed7c91c177
I20260806 10:55:42.864516 11233 ts_tablet_manager.cc:1939] T 3f6505a132e44610909d3eed7c91c177 P b9d43c880cfe4373bf15e9e00a59bd2f: Deleting consensus metadata
I20260806 10:55:42.867282 10678 catalog_manager.cc:4841] TS b9d43c880cfe4373bf15e9e00a59bd2f (127.30.218.3:35783): tablet 3f6505a132e44610909d3eed7c91c177 (table test-table [id=f2daa55abf62404dbc57a82a5c6d4d30]) successfully deleted
I20260806 10:55:43.016222 11232 ts_tablet_manager.cc:1918] T 3f6505a132e44610909d3eed7c91c177 P 893ec918c35548f4a22dd3eb40599640: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:55:43.016506 11232 log.cc:1196] T 3f6505a132e44610909d3eed7c91c177 P 893ec918c35548f4a22dd3eb40599640: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786013589554914-31592-0/minicluster-data/ts-0/wal/wals/3f6505a132e44610909d3eed7c91c177
I20260806 10:55:43.016850 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 10773
W20260806 10:55:43.034054 10678 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.218.1:39145 (error 108)
I20260806 10:55:43.034346 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 10916
I20260806 10:55:43.036369 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:43.035513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:39145 (local address 127.0.0.1:46746)
0806 10:55:43.036001 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0806 10:55:43.036133 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:39145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":240,"threads_started":1}
W20260806 10:55:43.036674 10678 catalog_manager.cc:4584] TS 893ec918c35548f4a22dd3eb40599640 (127.30.218.1:39145): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3f6505a132e44610909d3eed7c91c177: Network error: Client connection negotiation failed: client connection to 127.30.218.1:39145: connect: Connection refused (error 111)
Aug 06 10:55:43 dist-test-slave-49h3 krb5kdc[10211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.2@KRBTEST.COM
I20260806 10:55:43.050664 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:43.035231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33404 (local address 127.30.218.2:46527)
0806 10:55:43.035412 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 10:55:43.035418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:55:43.035465 (+ 47us) server_negotiation.cc:407] Connection header received
0806 10:55:43.035688 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:55:43.035694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:55:43.035790 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:55:43.035892 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:55:43.037417 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:43.038239 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:43.039678 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:43.040088 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:43.040214 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:55:43.044468 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:55:43.044484 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:55:43.044501 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:55:43.044552 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:55:43.047720 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:55:43.048446 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:55:43.048454 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:55:43.048456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:55:43.048612 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:55:43.048979 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:55:43.048983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:55:43.048985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:55:43.049371 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:55:43.049794 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0806 10:55:43.050024 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 10:55:43.050254 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 10:55:43.051314 11049 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:55:43.090461 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:43.089925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:39145 (local address 127.0.0.1:46752)
0806 10:55:43.090158 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0806 10:55:43.090284 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:39145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 10:55:43.091708 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:43.091268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:39145 (local address 127.0.0.1:46756)
0806 10:55:43.091441 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 10:55:43.091534 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:39145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 10:55:43.143818 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 10916
I20260806 10:55:43.152634 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:43.152019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:39145 (local address 127.0.0.1:46758)
0806 10:55:43.152296 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 10:55:43.152420 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:39145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260806 10:55:43.154053 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:43.153573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:39145 (local address 127.0.0.1:46764)
0806 10:55:43.153752 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 10:55:43.153858 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:39145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 10:55:43.162395 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 11061
Aug 06 10:55:43 dist-test-slave-49h3 krb5kdc[10211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.3@KRBTEST.COM
I20260806 10:55:43.180413 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:43.163301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41532 (local address 127.30.218.3:35783)
0806 10:55:43.163489 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 10:55:43.163496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:55:43.163522 (+ 26us) server_negotiation.cc:407] Connection header received
0806 10:55:43.163748 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:55:43.163753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:55:43.163844 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:55:43.163938 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:55:43.165333 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:43.166300 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:43.167911 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:55:43.168413 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:55:43.168552 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:55:43.173745 (+ 5193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:55:43.173761 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:55:43.173777 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:55:43.173836 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:55:43.177080 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:55:43.178084 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:55:43.178093 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:55:43.178095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:55:43.178291 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:55:43.178711 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:55:43.178715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:55:43.178718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:55:43.179122 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:55:43.179481 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0806 10:55:43.179782 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 10:55:43.180026 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 10:55:43.181110 11185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:55:43.234894 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:43.234371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:39145 (local address 127.0.0.1:46776)
0806 10:55:43.234640 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 10:55:43.234735 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:39145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260806 10:55:43.236122 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:43.235706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:39145 (local address 127.0.0.1:46790)
0806 10:55:43.235896 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0806 10:55:43.235973 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:39145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260806 10:55:43.308652 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 11061
I20260806 10:55:43.328534 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 10660
I20260806 10:55:43.329927 10723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:55:43.444412 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:55:43.443951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:39145 (local address 127.0.0.1:46798)
0806 10:55:43.444140 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 10:55:43.444254 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:39145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 10:55:43.444473 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 10660
I20260806 10:55:43.465924 31592 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 10:55:43.812690 31592 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 10:55:43.813076 31592 mini_ranger.cc:66] Stopping Ranger...
I20260806 10:55:44.160336 31592 mini_ranger.cc:68] Stopped Ranger
2026-08-06 10:55:44.160 UTC [10241] LOG: received smart shutdown request
2026-08-06 10:55:44.164 UTC [10241] LOG: background worker "logical replication launcher" (PID 10250) exited with exit code 1
2026-08-06 10:55:44.164 UTC [10245] LOG: shutting down
2026-08-06 10:55:44.315 UTC [10241] LOG: database system is shut down
2026-08-06T10:55:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51845 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:55:44 dist-test-slave-49h3 krb5kdc[11250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:55:44 dist-test-slave-49h3 krb5kdc[11250](info): set up 2 sockets
Aug 06 10:55:44 dist-test-slave-49h3 krb5kdc[11250](info): commencing operation
krb5kdc: starting...
W20260806 10:55:45.550638 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.003s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:55:45 dist-test-slave-49h3 krb5kdc[11250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:55:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:55:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.30.218.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.30.218.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.30.218.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/rangeradmin_127.30.218.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.30.218.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/rangeradmin_127.30.218.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.30.218.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.30.218.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.30.218.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/rangerlookup_127.30.218.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.30.218.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/rangerlookup_127.30.218.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.218.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.30.218.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.218.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/HTTP_127.30.218.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.218.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/HTTP_127.30.218.20@KRBTEST.COM.keytab.
I20260806 10:55:45.976274 31592 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskT4s7YX/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/postgres -l logfile start
2026-08-06 10:55:48.318 UTC [11280] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 10:55:48.318 UTC [11280] LOG: listening on IPv4 address "127.30.218.20", port 49871
2026-08-06 10:55:48.322 UTC [11280] LOG: listening on Unix socket "/tmp/.s.PGSQL.49871"
2026-08-06 10:55:48.346 UTC [11282] LOG: database system was shut down at 2026-08-06 10:55:47 UTC
2026-08-06 10:55:48.350 UTC [11280] LOG: database system is ready to accept connections
I20260806 10:55:49.285609 31592 mini_postgres.cc:96] Postgres bound to 49871
2026-08-06 10:55:49.310 UTC [11291] FATAL: database "slave" does not exist
127.30.218.20:49871 - accepting connections
I20260806 10:55:49.310997 31592 mini_ranger.cc:161] Starting Ranger...
I20260806 10:55:49.345093 31592 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 10:55:49.781821 31592 mini_ranger.cc:87] Created ranger Postgres database
I20260806 10:55:49.781939 31592 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ranger-admin
2026-08-06 10:55:50,081 [I] DB FLAVOR :POSTGRES
2026-08-06 10:55:50,082 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 10:55:50,082 [I] Checking connection..
2026-08-06 10:55:50,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:49871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 10:55:50,355 [I] Checking connection passed.
2026-08-06 10:55:50,356 [I] --------- Verifying version history table ---------
2026-08-06 10:55:50,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:49871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 10:55:50,626 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 10:55:50,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:49871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 10:55:50,872 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 10:55:50,872 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 10:55:50,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:49871/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 10:55:51.116 UTC [11356] WARNING: there is no transaction in progress
2026-08-06 10:55:51,123 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 10:55:51,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:49871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 10:55:51,366 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 10:55:51,366 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 10:55:51,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:49871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 10:55:51,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:49871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 10:55:51,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:49871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-49h3.c.gcp-upstream.internal','N') ;"
2026-08-06 10:55:52,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:49871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 10:55:52,411 [I] Table x_portal_user does not exist in database ranger
2026-08-06 10:55:52,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:49871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 10:55:52,670 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 10:55:52,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:49871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 10:55:52,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:49871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 10:55:53,198 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 10:55:53,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:49871/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 10:55:54.180 UTC [11466] WARNING: there is no transaction in progress
2026-08-06 10:55:54.198 UTC [11466] WARNING: there is no transaction in progress
2026-08-06 10:55:54.216 UTC [11466] WARNING: there is no transaction in progress
2026-08-06 10:55:54.233 UTC [11466] WARNING: there is no transaction in progress
2026-08-06 10:55:54.249 UTC [11466] WARNING: there is no transaction in progress
2026-08-06 10:55:54.269 UTC [11466] WARNING: there is no transaction in progress
2026-08-06 10:55:54.340 UTC [11466] WARNING: there is no transaction in progress
2026-08-06 10:55:54.347 UTC [11466] WARNING: there is no transaction in progress
2026-08-06 10:55:54.362 UTC [11466] WARNING: there is no transaction in progress
2026-08-06 10:55:54.372 UTC [11466] WARNING: there is no transaction in progress
2026-08-06 10:55:54.384 UTC [11466] WARNING: there is no transaction in progress
2026-08-06 10:55:54.400 UTC [11466] WARNING: there is no transaction in progress
2026-08-06 10:55:54.411 UTC [11466] WARNING: there is no transaction in progress
2026-08-06 10:55:54.418 UTC [11466] WARNING: there is no transaction in progress
2026-08-06 10:55:54.424 UTC [11466] WARNING: there is no transaction in progress
2026-08-06 10:55:54.431 UTC [11466] WARNING: there is no transaction in progress
2026-08-06 10:55:55,084 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 10:55:55,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:49871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 10:55:55,349 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 10:55:55,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:49871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 10:55:55,604 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 10:55:55,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:49871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 10:55:55,853 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 10:55:55,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:49871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 10:55:56,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:49871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 10:55:56,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:49871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-49h3.c.gcp-upstream.internal';"
2026-08-06 10:55:56,655 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 10:55:56,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:49871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 10:55:56,934 [I] DB_PATCHES have already been applied
I20260806 10:55:56.940940 31592 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ranger-admin:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskT4s7YX/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 10:55:56.941042 31592 mini_ranger.cc:193] Using host: 127.30.218.20
I20260806 10:55:56.963414 31592 mini_ranger.cc:227] Ranger admin URL: http://127.30.218.20:48205
Aug 06, 2026 10:55:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 10:55:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 10:55:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 10:55:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 10:55:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 10:55:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 10:55:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 10:55:59.106698 31592 mini_ranger.cc:160] Time spent starting Ranger: real 9.796s user 0.004s sys 0.078s
Aug 06, 2026 10:56:20 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 10:56:20 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 06, 2026 10:56:20 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 10:56:20 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 10:56:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 10:56:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 10:56:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 10:56:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 10:56:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 10:56:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 10:56:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 10:56:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 10:56:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 10:56:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 10:56:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 10:56:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 10:56:20 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 10:56:21 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 10:56:21.918066 31592 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.30.218.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.30.218.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.30.218.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/rangerkms_127.30.218.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.30.218.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/rangerkms_127.30.218.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.218.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.30.218.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.218.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/HTTP_127.30.218.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.218.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/HTTP_127.30.218.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 10:56:22 dist-test-slave-49h3 krb5kdc[11250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013782, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 10:56:22.168356 31592 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 10:56:22.196904 31592 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 10:56:23.009366 31592 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 10:56:23.009533 31592 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ranger-kms
I20260806 10:56:23.009595 31592 mini_ranger_kms.cc:227] Using postgres at 127.30.218.20:49871
2026-08-06 10:56:23,370 [I] DB FLAVOR :POSTGRES
2026-08-06 10:56:23,371 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 10:56:23,371 [I] Checking connection
2026-08-06 10:56:23,371 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:49871/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 10:56:23,626 [I] connection success
2026-08-06 10:56:23,626 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 10:56:23,626 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 10:56:23,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:49871/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 10:56:23,914 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 10:56:23,914 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 10:56:23,915 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 10:56:23,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.218.20:49871/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 10:56:24,227 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 10:56:24.780692 31592 mini_ranger_kms.cc:310] Created kms service
I20260806 10:56:25.033309 31592 mini_ranger_kms.cc:326] Created kudu user
I20260806 10:56:25.080525 31592 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 10:56:25 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013782, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.30.218.20@KRBTEST.COM
I20260806 10:56:25.501071 31592 mini_ranger_kms.cc:384] Added ranger policy
I20260806 10:56:25.501415 31592 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ranger-kms:/tmp/dist-test-taskT4s7YX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskT4s7YX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 10:56:25.501502 31592 mini_ranger_kms.cc:240] Using host: 127.30.218.20
I20260806 10:56:25.524657 31592 mini_ranger_kms.cc:276] Ranger KMS PID: 11652
I20260806 10:56:25.524801 31592 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.30.218.20:50737
Aug 06, 2026 10:56:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 10:56:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 10:56:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 10:56:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 10:56:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 10:56:27 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260806 10:56:27.696907 31592 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.529s user 0.076s sys 0.328s
I20260806 10:56:27.697130 31592 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 10:56:27.697296 31592 mini_ranger_kms.cc:401] 127.30.218.20:50737/kms/v1/keys
Aug 06 10:56:27 dist-test-slave-49h3 krb5kdc[11250](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786013787, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.218.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 06 10:56:28 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013787, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.218.20@KRBTEST.COM for HTTP/127.30.218.20@KRBTEST.COM
Aug 06 10:56:28 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013787, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.218.20@KRBTEST.COM for HTTP/127.30.218.20@KRBTEST.COM
Aug 06 10:56:31 dist-test-slave-49h3 krb5kdc[11250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:56:31.198244 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:37811
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:42659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:37811
--ranger_config_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.218.20:50737/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:56:31.560632 11703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:56:31.560981 11703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:56:31.561107 11703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:56:31.568933 11703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:56:31.569090 11703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:56:31.569139 11703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:56:31.569216 11703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:56:31.569278 11703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:56:31.580379 11703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:42659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/wal
--ranger_kms_url=127.30.218.20:50737/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:37811
--ranger_config_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:37811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:56:31.581907 11703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:56:31.583376 11703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:56:31.598249 11709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:31.598547 11711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:31.598790 11708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:31.599280 11703 server_base.cc:1034] running on GCE node
I20260806 10:56:31.600435 11703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:56:31.602669 11703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:56:31.603963 11703 hybrid_clock.cc:648] HybridClock initialized: now 1786013791603894 us; error 59 us; skew 500 ppm
I20260806 10:56:31.604470 11703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:56:31 dist-test-slave-49h3 krb5kdc[11250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013791, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:31.613866 11703 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:56:31.615722 11703 webserver.cc:460] Webserver started at http://127.30.218.62:40593/ using document root <none> and password file <none>
I20260806 10:56:31.616462 11703 fs_manager.cc:359] Metadata directory not provided
I20260806 10:56:31.616582 11703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:56:31.616912 11703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 10:56:31 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013791, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for HTTP/127.30.218.20@KRBTEST.COM
I20260806 10:56:31.672050 11703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "0d10df381f8c40279a3d7696621dabbd"
format_stamp: "Formatted at 2026-08-06 10:56:31 on dist-test-slave-49h3"
server_key: "612dd5c38eca1cdba52437e0fd499d62"
server_key_iv: "6af5b4e14c22b1dad83acd1f68ef32d3"
server_key_version: "kuduclusterkey@0"
I20260806 10:56:31.673038 11703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "0d10df381f8c40279a3d7696621dabbd"
format_stamp: "Formatted at 2026-08-06 10:56:31 on dist-test-slave-49h3"
server_key: "612dd5c38eca1cdba52437e0fd499d62"
server_key_iv: "6af5b4e14c22b1dad83acd1f68ef32d3"
server_key_version: "kuduclusterkey@0"
I20260806 10:56:31.679077 11703 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260806 10:56:31.697283 11718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:31.698951 11703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 10:56:31.699167 11703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "0d10df381f8c40279a3d7696621dabbd"
format_stamp: "Formatted at 2026-08-06 10:56:31 on dist-test-slave-49h3"
server_key: "612dd5c38eca1cdba52437e0fd499d62"
server_key_iv: "6af5b4e14c22b1dad83acd1f68ef32d3"
server_key_version: "kuduclusterkey@0"
I20260806 10:56:31.699414 11703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:56:31.717167 11703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:56:31.721825 11703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:56:31.722167 11703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:56:31.741781 11703 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:37811
I20260806 10:56:31.741782 11769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:37811 every 8 connection(s)
I20260806 10:56:31.743335 11703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:56:31.744230 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 11703
I20260806 10:56:31.744432 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:56:31.745249 31592 external_mini_cluster.cc:1428] Setting key 4b07ffe9a4e036f18f0e1dcad763b748
I20260806 10:56:31.749363 11770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:56:31.765110 11770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd: Bootstrap starting.
I20260806 10:56:31.769435 11770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd: Neither blocks nor log segments found. Creating new log.
Aug 06 10:56:31 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:56:31.770639 11770 log.cc:824] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd: Log is configured to *not* fsync() on all Append() calls
I20260806 10:56:31.773924 11770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd: No bootstrap required, opened a new log
I20260806 10:56:31.782059 11773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:31.747746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53916 (local address 127.30.218.62:37811)
0806 10:56:31.754164 (+ 6418us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:31.754183 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:31.754890 (+ 707us) server_negotiation.cc:407] Connection header received
0806 10:56:31.757243 (+ 2353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:31.757281 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:31.757967 (+ 686us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:31.758460 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:31.760243 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:31.762375 (+ 2132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:31.764378 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:31.765543 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:31.765742 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:31.771309 (+ 5567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:31.771321 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:31.771342 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:31.771403 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:31.776521 (+ 5118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:31.777552 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:31.777561 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:31.777570 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:31.777798 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:31.778242 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:31.778246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:31.778250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:31.779032 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:31.779333 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:31.779825 (+ 492us) server_negotiation.cc:299] Negotiation successful
0806 10:56:31.780514 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6096,"thread_start_us":5819,"threads_started":1}
I20260806 10:56:31.782793 11770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d10df381f8c40279a3d7696621dabbd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 37811 } }
I20260806 10:56:31.783175 11770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:56:31.783337 11770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d10df381f8c40279a3d7696621dabbd, State: Initialized, Role: FOLLOWER
I20260806 10:56:31.784020 11770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d10df381f8c40279a3d7696621dabbd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 37811 } }
I20260806 10:56:31.784276 11770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:56:31.784441 11770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:56:31.784623 11770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:56:31.787029 11770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d10df381f8c40279a3d7696621dabbd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 37811 } }
I20260806 10:56:31.787529 11770 leader_election.cc:302] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d10df381f8c40279a3d7696621dabbd; no voters:
I20260806 10:56:31.788148 11770 leader_election.cc:288] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:56:31.788290 11775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:56:31.791352 11770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:56:31.795198 11775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [term 1 LEADER]: Becoming Leader. State: Replica: 0d10df381f8c40279a3d7696621dabbd, State: Running, Role: LEADER
I20260806 10:56:31.795758 11775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d10df381f8c40279a3d7696621dabbd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 37811 } }
I20260806 10:56:31.802078 11777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d10df381f8c40279a3d7696621dabbd. Latest consensus state: current_term: 1 leader_uuid: "0d10df381f8c40279a3d7696621dabbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d10df381f8c40279a3d7696621dabbd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 37811 } } }
I20260806 10:56:31.802464 11777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [sys.catalog]: This master's current role is: LEADER
I20260806 10:56:31.802999 11776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d10df381f8c40279a3d7696621dabbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d10df381f8c40279a3d7696621dabbd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 37811 } } }
I20260806 10:56:31.804445 11776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [sys.catalog]: This master's current role is: LEADER
I20260806 10:56:31.811038 11785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:56:31.811790 11770 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 10:56:31.828171 11785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:56:31.841909 11785 catalog_manager.cc:1347] Generated new cluster ID: 32a4eaf06b8b479bba3d541322d8fa98
I20260806 10:56:31.842043 11785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:56:31.861195 11785 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:56:31.863267 11785 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:56:31.876487 11785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd: Generated new TSK 0
I20260806 10:56:31.877993 11785 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
I20260806 10:56:32.765813 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.218.20:50737/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:37811
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:42659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 10:56:33 dist-test-slave-49h3 krb5kdc[11250](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786013793, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 10:56:33.253733 11819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:56:33.254283 11819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:56:33.254473 11819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:56:33.266855 11819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:56:33.267130 11819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:56:33.267397 11819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:56:33.284255 11819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:42659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.30.218.20:50737/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:37811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:56:33.286378 11819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:56:33.288338 11819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:56:33.305245 11829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:33.305294 11830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:33.305764 11832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:33.306351 11819 server_base.cc:1034] running on GCE node
I20260806 10:56:33.306988 11819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:56:33.308044 11819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:56:33.309258 11819 hybrid_clock.cc:648] HybridClock initialized: now 1786013793309196 us; error 57 us; skew 500 ppm
I20260806 10:56:33.309631 11819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:56:33 dist-test-slave-49h3 krb5kdc[11250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013793, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:33.313293 11819 init.cc:377] Logged in from keytab as kudu/127.30.218.1@KRBTEST.COM (short username kudu)
I20260806 10:56:33.315434 11819 webserver.cc:460] Webserver started at http://127.30.218.1:43777/ using document root <none> and password file <none>
I20260806 10:56:33.316562 11819 fs_manager.cc:359] Metadata directory not provided
I20260806 10:56:33.316749 11819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:56:33.317188 11819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 10:56:33 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013793, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for HTTP/127.30.218.20@KRBTEST.COM
I20260806 10:56:33.344728 11819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "83af05218d3d419bb47d69bf7944d578"
format_stamp: "Formatted at 2026-08-06 10:56:33 on dist-test-slave-49h3"
server_key: "cc947fac8e0bf2d3967a0bc85d3d2e9e"
server_key_iv: "7fe0a1e65bc1cbc224f7d95b8543f003"
server_key_version: "kuduclusterkey@0"
I20260806 10:56:33.347347 11819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "83af05218d3d419bb47d69bf7944d578"
format_stamp: "Formatted at 2026-08-06 10:56:33 on dist-test-slave-49h3"
server_key: "cc947fac8e0bf2d3967a0bc85d3d2e9e"
server_key_iv: "7fe0a1e65bc1cbc224f7d95b8543f003"
server_key_version: "kuduclusterkey@0"
I20260806 10:56:33.372522 11819 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.011s sys 0.012s
I20260806 10:56:33.404721 11839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:33.407070 11819 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.005s sys 0.000s
I20260806 10:56:33.407423 11819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "83af05218d3d419bb47d69bf7944d578"
format_stamp: "Formatted at 2026-08-06 10:56:33 on dist-test-slave-49h3"
server_key: "cc947fac8e0bf2d3967a0bc85d3d2e9e"
server_key_iv: "7fe0a1e65bc1cbc224f7d95b8543f003"
server_key_version: "kuduclusterkey@0"
I20260806 10:56:33.407814 11819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:56:33.478879 11819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:56:33.484905 11819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:56:33.485327 11819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:56:33.514387 11819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:56:33.516772 11819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:56:33.516898 11819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:33.517047 11819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:56:33.517114 11819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 10:56:33 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013793, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for HTTP/127.30.218.20@KRBTEST.COM
I20260806 10:56:33.639611 11819 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:45481
I20260806 10:56:33.639629 11953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:45481 every 8 connection(s)
I20260806 10:56:33.641041 11819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:56:33.647034 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 11819
I20260806 10:56:33.647289 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:56:33.648188 31592 external_mini_cluster.cc:1428] Setting key e6be5586a421d8f9bc5021e2771704b4
Aug 06 10:56:33 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013793, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:56:33.692696 11957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:33.645051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:37811 (local address 127.30.218.1:40063)
0806 10:56:33.658677 (+ 13626us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:33.658741 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:33.661486 (+ 2745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:33.664832 (+ 3346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:33.664846 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:33.665644 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:56:33.666642 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:33.666675 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:33.668368 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:33.668373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:33.670312 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:33.670332 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:33.671464 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:33.671471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:33.671785 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:33.679225 (+ 7440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:56:33.679265 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:56:33.682449 (+ 3184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:56:33.687066 (+ 4617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:33.687079 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:33.687115 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:33.687872 (+ 757us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:33.688466 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:33.688472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:33.688476 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:33.688774 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:33.689572 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:56:33.689583 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:56:33.690108 (+ 525us) client_negotiation.cc:769] Sending connection context
0806 10:56:33.690456 (+ 348us) client_negotiation.cc:240] Negotiation successful
0806 10:56:33.691101 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13263,"thread_start_us":13002,"threads_started":1}
WARNING: no policy specified for kudu/127.30.218.2@KRBTEST.COM; defaulting to no policy
I20260806 10:56:33.694692 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:33.644270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:40063 (local address 127.30.218.62:37811)
0806 10:56:33.657990 (+ 13720us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:33.657998 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:33.662153 (+ 4155us) server_negotiation.cc:407] Connection header received
0806 10:56:33.664231 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:33.664237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:33.664363 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:33.664499 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:33.666900 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:33.668165 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:33.670520 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:33.671154 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:33.671324 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:33.682893 (+ 11569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:33.682902 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:33.682907 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:33.682972 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:33.686814 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:33.688079 (+ 1265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:33.688085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:33.688100 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:33.688306 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:33.688948 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:33.688953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:33.688957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:33.689370 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:33.693928 (+ 4558us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:33.694105 (+ 177us) server_negotiation.cc:299] Negotiation successful
0806 10:56:33.694405 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13546,"thread_start_us":13421,"threads_started":1}
I20260806 10:56:33.696434 11954 heartbeater.cc:347] Connected to a master server at 127.30.218.62:37811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.2@KRBTEST.COM" created.
I20260806 10:56:33.696852 11954 heartbeater.cc:464] Registering TS with master...
I20260806 10:56:33.697904 11954 heartbeater.cc:511] Master 127.30.218.62:37811 requested a full tablet report, sending...
I20260806 10:56:33.701673 11735 ts_manager.cc:194] Registered new tserver with Master: 83af05218d3d419bb47d69bf7944d578 (127.30.218.1:45481)
I20260806 10:56:33.703960 11735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.1@KRBTEST.COM'} at 127.30.218.1:40063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
I20260806 10:56:33.736680 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.218.20:50737/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:37811
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:42659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 10:56:33 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013793, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for HTTP/127.30.218.20@KRBTEST.COM
W20260806 10:56:34.229367 11960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:56:34.229951 11960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:56:34.230103 11960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:56:34.238418 11960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:56:34.238569 11960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:56:34.238762 11960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:56:34.254181 11960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:42659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.30.218.20:50737/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:37811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:56:34.256328 11960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:56:34.258015 11960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:56:34.273272 11972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:34.273314 11974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:34.273742 11960 server_base.cc:1034] running on GCE node
W20260806 10:56:34.274230 11977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:34.274945 11960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:56:34.276023 11960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:56:34.277256 11960 hybrid_clock.cc:648] HybridClock initialized: now 1786013794277207 us; error 44 us; skew 500 ppm
I20260806 10:56:34.277621 11960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:56:34 dist-test-slave-49h3 krb5kdc[11250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013794, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:34.281049 11960 init.cc:377] Logged in from keytab as kudu/127.30.218.2@KRBTEST.COM (short username kudu)
I20260806 10:56:34.282898 11960 webserver.cc:460] Webserver started at http://127.30.218.2:38931/ using document root <none> and password file <none>
I20260806 10:56:34.283667 11960 fs_manager.cc:359] Metadata directory not provided
I20260806 10:56:34.283788 11960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:56:34.284111 11960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 10:56:34 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013794, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for HTTP/127.30.218.20@KRBTEST.COM
I20260806 10:56:34.302091 11960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "3ff86ea049094be6ba7d367280fb3693"
format_stamp: "Formatted at 2026-08-06 10:56:34 on dist-test-slave-49h3"
server_key: "0a2a8309151f5954411b64b0cae101db"
server_key_iv: "d4b5cdb79306f81e22f978ac8d940776"
server_key_version: "kuduclusterkey@0"
I20260806 10:56:34.303155 11960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "3ff86ea049094be6ba7d367280fb3693"
format_stamp: "Formatted at 2026-08-06 10:56:34 on dist-test-slave-49h3"
server_key: "0a2a8309151f5954411b64b0cae101db"
server_key_iv: "d4b5cdb79306f81e22f978ac8d940776"
server_key_version: "kuduclusterkey@0"
I20260806 10:56:34.310050 11960 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260806 10:56:34.329267 11984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:34.330952 11960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:56:34.331151 11960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "3ff86ea049094be6ba7d367280fb3693"
format_stamp: "Formatted at 2026-08-06 10:56:34 on dist-test-slave-49h3"
server_key: "0a2a8309151f5954411b64b0cae101db"
server_key_iv: "d4b5cdb79306f81e22f978ac8d940776"
server_key_version: "kuduclusterkey@0"
I20260806 10:56:34.331434 11960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:56:34.388120 11960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:56:34.392961 11960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:56:34.393275 11960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:56:34.394490 11960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:56:34.396153 11960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:56:34.396248 11960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:34.396369 11960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:56:34.396431 11960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:34.448890 11960 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:34293
I20260806 10:56:34.448913 12097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:34293 every 8 connection(s)
I20260806 10:56:34.450462 11960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:56:34.454036 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 11960
I20260806 10:56:34.454237 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:56:34.454877 31592 external_mini_cluster.cc:1428] Setting key 2000a9233f35737e6b314e9ae0cb2bf1
Aug 06 10:56:34 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013794, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:56:34.488198 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:34.456390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:58001 (local address 127.30.218.62:37811)
0806 10:56:34.457002 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:34.457009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:34.459981 (+ 2972us) server_negotiation.cc:407] Connection header received
0806 10:56:34.463610 (+ 3629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:34.463617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:34.463757 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:34.463848 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:34.466613 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:34.467786 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:34.470431 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:34.471101 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:34.471234 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:34.476238 (+ 5004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:34.476246 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:34.476251 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:34.476306 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:34.480460 (+ 4154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:34.481744 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:34.481751 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:34.481754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:34.481938 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:34.482560 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:34.482565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:34.482568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:34.482983 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:34.483187 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:34.484235 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0806 10:56:34.484496 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":2428,"threads_started":1}
I20260806 10:56:34.490914 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:34.458452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:37811 (local address 127.30.218.2:58001)
0806 10:56:34.459416 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:34.459477 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:34.463175 (+ 3698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:34.464547 (+ 1372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:34.464562 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:34.465281 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:56:34.466353 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:34.466394 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:34.468042 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:34.468047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:34.470220 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:34.470240 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:34.471680 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:34.471691 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:34.472043 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:34.472865 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:56:34.472897 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:56:34.475894 (+ 2997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:56:34.480733 (+ 4839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:34.480746 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:34.480768 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:34.481547 (+ 779us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:34.482107 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:34.482112 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:34.482116 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:34.482419 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:34.483286 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:56:34.483295 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:56:34.483784 (+ 489us) client_negotiation.cc:769] Sending connection context
0806 10:56:34.488818 (+ 5034us) client_negotiation.cc:240] Negotiation successful
0806 10:56:34.489516 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":394,"threads_started":1}
I20260806 10:56:34.493073 12098 heartbeater.cc:347] Connected to a master server at 127.30.218.62:37811
I20260806 10:56:34.493402 12098 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.30.218.3@KRBTEST.COM; defaulting to no policy
I20260806 10:56:34.494391 12098 heartbeater.cc:511] Master 127.30.218.62:37811 requested a full tablet report, sending...
I20260806 10:56:34.496284 11735 ts_manager.cc:194] Registered new tserver with Master: 3ff86ea049094be6ba7d367280fb3693 (127.30.218.2:34293)
I20260806 10:56:34.497347 11735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.2@KRBTEST.COM'} at 127.30.218.2:58001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
I20260806 10:56:34.538825 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.218.20:50737/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:37811
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:42659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:56:34.708562 11954 heartbeater.cc:503] Master 127.30.218.62:37811 was elected leader, sending a full tablet report...
W20260806 10:56:34.945844 12104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:56:34.946295 12104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:56:34.946449 12104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:56:34.956274 12104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:56:34.956422 12104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:56:34.956601 12104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:56:34.968773 12104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:42659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.30.218.20:50737/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:37811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:56:34.970279 12104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:56:34.971660 12104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:56:34.985263 12109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:34.985263 12110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:34.985351 12112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:34.986430 12104 server_base.cc:1034] running on GCE node
I20260806 10:56:34.987078 12104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:56:34.988098 12104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:56:34.989310 12104 hybrid_clock.cc:648] HybridClock initialized: now 1786013794989253 us; error 53 us; skew 500 ppm
I20260806 10:56:34.989696 12104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:56:34 dist-test-slave-49h3 krb5kdc[11250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013794, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:34.993014 12104 init.cc:377] Logged in from keytab as kudu/127.30.218.3@KRBTEST.COM (short username kudu)
I20260806 10:56:34.994699 12104 webserver.cc:460] Webserver started at http://127.30.218.3:40993/ using document root <none> and password file <none>
I20260806 10:56:34.995421 12104 fs_manager.cc:359] Metadata directory not provided
I20260806 10:56:34.995548 12104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:56:34.995862 12104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 10:56:34 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013794, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for HTTP/127.30.218.20@KRBTEST.COM
I20260806 10:56:35.012405 12104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "762824a02e294c1897162b92e75fa8ea"
format_stamp: "Formatted at 2026-08-06 10:56:35 on dist-test-slave-49h3"
server_key: "a20d887ca29e851563cadd82a59ab3c0"
server_key_iv: "68456fea9d731bca66de8a1d35a1315e"
server_key_version: "kuduclusterkey@0"
I20260806 10:56:35.013438 12104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "762824a02e294c1897162b92e75fa8ea"
format_stamp: "Formatted at 2026-08-06 10:56:35 on dist-test-slave-49h3"
server_key: "a20d887ca29e851563cadd82a59ab3c0"
server_key_iv: "68456fea9d731bca66de8a1d35a1315e"
server_key_version: "kuduclusterkey@0"
I20260806 10:56:35.019851 12104 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 10:56:35.035727 12119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:35.037396 12104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260806 10:56:35.037634 12104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "762824a02e294c1897162b92e75fa8ea"
format_stamp: "Formatted at 2026-08-06 10:56:35 on dist-test-slave-49h3"
server_key: "a20d887ca29e851563cadd82a59ab3c0"
server_key_iv: "68456fea9d731bca66de8a1d35a1315e"
server_key_version: "kuduclusterkey@0"
I20260806 10:56:35.037874 12104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:56:35.050587 12104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:56:35.055090 12104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:56:35.055394 12104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:56:35.056484 12104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:56:35.058036 12104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:56:35.058115 12104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:35.058219 12104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:56:35.058271 12104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:35.090165 12104 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:44267
I20260806 10:56:35.090204 12234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:44267 every 8 connection(s)
I20260806 10:56:35.091480 12104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:56:35.095009 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 12104
I20260806 10:56:35.095201 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:56:35.095767 31592 external_mini_cluster.cc:1428] Setting key 8827a25688b4af3f49e0f7a88fb099ea
Aug 06 10:56:35 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013794, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:56:35.113699 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:35.094268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:53857 (local address 127.30.218.62:37811)
0806 10:56:35.094868 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:35.094874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:35.095998 (+ 1124us) server_negotiation.cc:407] Connection header received
0806 10:56:35.098363 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:35.098368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:35.098479 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:35.098606 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:35.100261 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:35.100990 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:35.102775 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:35.103244 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:35.103344 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:35.107001 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:35.107006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:35.107009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:35.107048 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:35.109980 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:35.111071 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:35.111076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:35.111079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:35.111272 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:35.111842 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:35.111846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:35.111848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:35.112244 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:35.112387 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:35.113141 (+ 754us) server_negotiation.cc:299] Negotiation successful
0806 10:56:35.113364 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":279,"threads_started":1}
I20260806 10:56:35.114665 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:35.094887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:37811 (local address 127.30.218.3:53857)
0806 10:56:35.095746 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:35.095837 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:35.098050 (+ 2213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:35.098829 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:35.098838 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:35.099437 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:56:35.100083 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:35.100103 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:35.101140 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:35.101143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:35.102639 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:35.102653 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:35.103453 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:35.103458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:35.103719 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:35.104464 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:56:35.104490 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:56:35.106756 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:56:35.110195 (+ 3439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:35.110206 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:35.110227 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:35.110906 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:35.111410 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:35.111415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:35.111418 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:35.111694 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:35.112375 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:56:35.112381 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:56:35.112789 (+ 408us) client_negotiation.cc:769] Sending connection context
0806 10:56:35.113030 (+ 241us) client_negotiation.cc:240] Negotiation successful
0806 10:56:35.113529 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":318,"threads_started":1}
I20260806 10:56:35.116268 12235 heartbeater.cc:347] Connected to a master server at 127.30.218.62:37811
I20260806 10:56:35.116612 12235 heartbeater.cc:464] Registering TS with master...
I20260806 10:56:35.117379 12235 heartbeater.cc:511] Master 127.30.218.62:37811 requested a full tablet report, sending...
I20260806 10:56:35.119205 11735 ts_manager.cc:194] Registered new tserver with Master: 762824a02e294c1897162b92e75fa8ea (127.30.218.3:44267)
I20260806 10:56:35.120082 11735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.3@KRBTEST.COM'} at 127.30.218.3:53857
I20260806 10:56:35.124899 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:56:35.126878 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 11819
Aug 06 10:56:35 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.1@KRBTEST.COM
I20260806 10:56:35.145565 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:35.128047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57176 (local address 127.30.218.1:45481)
0806 10:56:35.128685 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:35.128702 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:35.128729 (+ 27us) server_negotiation.cc:407] Connection header received
0806 10:56:35.128792 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:35.128798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:35.129027 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:35.129233 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:35.130730 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:35.131812 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:35.133591 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:35.134228 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:35.134340 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:35.139072 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:35.139089 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:35.139106 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:35.139161 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:35.142286 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:35.143221 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:35.143230 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:35.143232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:35.143414 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:35.143830 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:35.143834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:35.143836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:35.144213 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:35.144523 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:35.144851 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 10:56:35.145143 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":310,"threads_started":1}
I20260806 10:56:35.146942 11949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:56:35.233094 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 11819
I20260806 10:56:35.250077 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 11960
Aug 06 10:56:35 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.2@KRBTEST.COM
I20260806 10:56:35.267467 12242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:35.251091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37502 (local address 127.30.218.2:34293)
0806 10:56:35.251725 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:35.251732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:35.251759 (+ 27us) server_negotiation.cc:407] Connection header received
0806 10:56:35.251823 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:35.251830 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:35.252057 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:35.252201 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:35.253573 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:35.254641 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:35.256010 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:35.256610 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:35.256726 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:35.261435 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:35.261478 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:35.261504 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:35.261556 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:35.264599 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:35.265416 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:35.265423 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:35.265426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:35.265636 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:35.265977 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:35.265980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:35.265981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:35.266362 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:35.266616 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:35.266848 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 10:56:35.267066 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":354,"threads_started":1}
I20260806 10:56:35.268833 12093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:56:35.353120 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 11960
I20260806 10:56:35.372759 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 12104
Aug 06 10:56:35 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.3@KRBTEST.COM
I20260806 10:56:35.392390 12244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:35.373980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49930 (local address 127.30.218.3:44267)
0806 10:56:35.374692 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:35.374700 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:35.374731 (+ 31us) server_negotiation.cc:407] Connection header received
0806 10:56:35.374795 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:35.374802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:35.375014 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:35.375160 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:35.376730 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:35.378187 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:35.379852 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:35.380643 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:35.380784 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:35.385884 (+ 5100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:35.385904 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:35.385923 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:35.385980 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:35.389083 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:35.390131 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:35.390154 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:35.390157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:35.390329 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:35.390717 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:35.390721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:35.390723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:35.391161 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:35.391479 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:35.391838 (+ 359us) server_negotiation.cc:299] Negotiation successful
0806 10:56:35.392068 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":388,"threads_started":1}
I20260806 10:56:35.393774 12230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:56:35.478368 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 12104
I20260806 10:56:35.494730 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 11703
I20260806 10:56:35.496002 11765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:56:35.605482 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 11703
I20260806 10:56:35.632663 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:37811
--webserver_interface=127.30.218.62
--webserver_port=40593
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:42659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:37811
--ranger_config_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.218.20:50737/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.218.20:50737/kms with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:56:36.014909 12247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:56:36.015265 12247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:56:36.015357 12247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:56:36.023102 12247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:56:36.023239 12247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:56:36.023288 12247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:56:36.023345 12247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:56:36.023394 12247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:56:36.034369 12247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:42659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.218.20:50737/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:37811
--ranger_config_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:37811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=40593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:56:36.035766 12247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:56:36.037147 12247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:56:36.049818 12256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:36.049743 12253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:36.049783 12254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:36.050905 12247 server_base.cc:1034] running on GCE node
I20260806 10:56:36.051842 12247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:56:36.053417 12247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:56:36.054664 12247 hybrid_clock.cc:648] HybridClock initialized: now 1786013796054604 us; error 62 us; skew 500 ppm
I20260806 10:56:36.055027 12247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:56:36 dist-test-slave-49h3 krb5kdc[11250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013796, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:36.058393 12247 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:56:36.060113 12247 webserver.cc:460] Webserver started at http://127.30.218.62:40593/ using document root <none> and password file <none>
I20260806 10:56:36.060799 12247 fs_manager.cc:359] Metadata directory not provided
I20260806 10:56:36.060920 12247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 10:56:36 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013796, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for HTTP/127.30.218.20@KRBTEST.COM
I20260806 10:56:36.141180 12247 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 10:56:36.144327 12264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:36.145808 12247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:56:36.145983 12247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "0d10df381f8c40279a3d7696621dabbd"
format_stamp: "Formatted at 2026-08-06 10:56:31 on dist-test-slave-49h3"
server_key: "612dd5c38eca1cdba52437e0fd499d62"
server_key_iv: "6af5b4e14c22b1dad83acd1f68ef32d3"
server_key_version: "kuduclusterkey@0"
I20260806 10:56:36.146945 12247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:56:36.161760 12247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:56:36.166010 12247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:56:36.166343 12247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:56:36.185566 12247 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:37811
I20260806 10:56:36.185586 12316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:37811 every 8 connection(s)
I20260806 10:56:36.187258 12247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:56:36.192373 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 12247
I20260806 10:56:36.193549 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.218.20:50737/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:45481
--local_ip_for_outbound_sockets=127.30.218.1
--tserver_master_addrs=127.30.218.62:37811
--webserver_port=43777
--webserver_interface=127.30.218.1
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:42659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.218.20:50737/kms with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:56:36.196163 12317 sys_catalog.cc:263] Verifying existing consensus state
I20260806 10:56:36.201071 12317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd: Bootstrap starting.
I20260806 10:56:36.222978 12317 log.cc:824] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd: Log is configured to *not* fsync() on all Append() calls
I20260806 10:56:36.233717 12317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 10:56:36.234318 12317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd: Bootstrap complete.
I20260806 10:56:36.244076 12317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d10df381f8c40279a3d7696621dabbd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 37811 } }
I20260806 10:56:36.245036 12317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d10df381f8c40279a3d7696621dabbd, State: Initialized, Role: FOLLOWER
I20260806 10:56:36.245848 12317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d10df381f8c40279a3d7696621dabbd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 37811 } }
I20260806 10:56:36.246112 12317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:56:36.246248 12317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:56:36.246428 12317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [term 1 FOLLOWER]: Advancing to term 2
I20260806 10:56:36.251549 12317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d10df381f8c40279a3d7696621dabbd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 37811 } }
I20260806 10:56:36.252190 12317 leader_election.cc:302] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d10df381f8c40279a3d7696621dabbd; no voters:
I20260806 10:56:36.252853 12317 leader_election.cc:288] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 10:56:36.253034 12321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [term 2 FOLLOWER]: Leader election won for term 2
I20260806 10:56:36.254228 12321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [term 2 LEADER]: Becoming Leader. State: Replica: 0d10df381f8c40279a3d7696621dabbd, State: Running, Role: LEADER
I20260806 10:56:36.254904 12321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d10df381f8c40279a3d7696621dabbd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 37811 } }
I20260806 10:56:36.255266 12317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:56:36.257805 12323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d10df381f8c40279a3d7696621dabbd. Latest consensus state: current_term: 2 leader_uuid: "0d10df381f8c40279a3d7696621dabbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d10df381f8c40279a3d7696621dabbd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 37811 } } }
I20260806 10:56:36.257836 12322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0d10df381f8c40279a3d7696621dabbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d10df381f8c40279a3d7696621dabbd" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 37811 } } }
I20260806 10:56:36.258246 12323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [sys.catalog]: This master's current role is: LEADER
I20260806 10:56:36.258250 12322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd [sys.catalog]: This master's current role is: LEADER
I20260806 10:56:36.270179 12327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:56:36.276454 12327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:56:36.278177 12327 catalog_manager.cc:1259] Loaded cluster ID: 32a4eaf06b8b479bba3d541322d8fa98
I20260806 10:56:36.278395 12327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:56:36.283366 12327 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:56:36.285187 12327 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd: Loaded TSK: 0
I20260806 10:56:36.286924 12327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:56:36.293475 12317 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260806 10:56:36.632665 12319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:56:36.633179 12319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:56:36.633409 12319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:56:36.646003 12319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:56:36.646384 12319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:56:36.646695 12319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:56:36.663545 12319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:42659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.218.20:50737/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:45481
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=43777
--tserver_master_addrs=127.30.218.62:37811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:56:36.665675 12319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:56:36.667652 12319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:56:36.701598 12356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:36.705632 12355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:36.710789 12358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:36.711086 12319 server_base.cc:1034] running on GCE node
I20260806 10:56:36.712244 12319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:56:36.713873 12319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:56:36.715471 12319 hybrid_clock.cc:648] HybridClock initialized: now 1786013796715388 us; error 79 us; skew 500 ppm
I20260806 10:56:36.715972 12319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:56:36 dist-test-slave-49h3 krb5kdc[11250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013796, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:36.726261 12319 init.cc:377] Logged in from keytab as kudu/127.30.218.1@KRBTEST.COM (short username kudu)
I20260806 10:56:36.739576 12319 webserver.cc:460] Webserver started at http://127.30.218.1:43777/ using document root <none> and password file <none>
I20260806 10:56:36.740907 12319 fs_manager.cc:359] Metadata directory not provided
I20260806 10:56:36.741142 12319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 10:56:36 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013796, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for HTTP/127.30.218.20@KRBTEST.COM
I20260806 10:56:36.797660 12319 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 10:56:36.812801 12366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:36.815740 12319 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.007s sys 0.008s
I20260806 10:56:36.816147 12319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "83af05218d3d419bb47d69bf7944d578"
format_stamp: "Formatted at 2026-08-06 10:56:33 on dist-test-slave-49h3"
server_key: "cc947fac8e0bf2d3967a0bc85d3d2e9e"
server_key_iv: "7fe0a1e65bc1cbc224f7d95b8543f003"
server_key_version: "kuduclusterkey@0"
I20260806 10:56:36.818013 12319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:56:36.868764 12319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:56:36.874095 12319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:56:36.874433 12319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:56:36.875651 12319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:56:36.880434 12319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:56:36.880578 12319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:36.880733 12319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:56:36.880810 12319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:36.943039 12319 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:45481
I20260806 10:56:36.943112 12480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:45481 every 8 connection(s)
I20260806 10:56:36.945160 12319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:56:36.950898 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 12319
I20260806 10:56:36.951843 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.218.20:50737/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:34293
--local_ip_for_outbound_sockets=127.30.218.2
--tserver_master_addrs=127.30.218.62:37811
--webserver_port=38931
--webserver_interface=127.30.218.2
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:42659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.218.20:50737/kms with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 10:56:36 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013796, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:56:37.002053 12484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:36.948795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:37811 (local address 127.30.218.1:56093)
0806 10:56:36.967484 (+ 18689us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:36.967561 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:36.971394 (+ 3833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:36.974940 (+ 3546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:36.974953 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:36.975744 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:56:36.976715 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:36.976751 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:36.980127 (+ 3376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:36.980135 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:36.982254 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:36.982279 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:36.984038 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:36.984050 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:36.984376 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:36.985262 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:56:36.985304 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:56:36.990963 (+ 5659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:56:36.996137 (+ 5174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:36.996149 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:36.996169 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:36.996886 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:36.997600 (+ 714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:36.997606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:36.997611 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:36.997907 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:36.999023 (+ 1116us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:56:36.999032 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:56:36.999490 (+ 458us) client_negotiation.cc:769] Sending connection context
0806 10:56:36.999822 (+ 332us) client_negotiation.cc:240] Negotiation successful
0806 10:56:37.000482 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18234,"thread_start_us":20540,"threads_started":1}
I20260806 10:56:37.005995 12483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:36.948794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:56093 (local address 127.30.218.62:37811)
0806 10:56:36.972407 (+ 23613us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:36.972429 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:36.972482 (+ 53us) server_negotiation.cc:407] Connection header received
0806 10:56:36.972980 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:36.973001 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:36.973628 (+ 627us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:36.974124 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:36.977763 (+ 3639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:36.979844 (+ 2081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:36.982529 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:36.983697 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:36.983937 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:36.991461 (+ 7524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:36.991474 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:36.991499 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:36.991579 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:36.995820 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:36.997128 (+ 1308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:36.997140 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:36.997148 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:36.997423 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:36.998074 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:36.998080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:36.998084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:36.998854 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:37.003600 (+ 4746us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:37.003954 (+ 354us) server_negotiation.cc:299] Negotiation successful
0806 10:56:37.004494 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23296,"thread_start_us":23060,"threads_started":1}
I20260806 10:56:37.010427 12481 heartbeater.cc:347] Connected to a master server at 127.30.218.62:37811
I20260806 10:56:37.010811 12481 heartbeater.cc:464] Registering TS with master...
I20260806 10:56:37.011777 12481 heartbeater.cc:511] Master 127.30.218.62:37811 requested a full tablet report, sending...
W20260806 10:56:37.015916 12481 heartbeater.cc:644] Failed to heartbeat to 127.30.218.62:37811 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 10:56:37.016191 12481 heartbeater.cc:464] Registering TS with master...
I20260806 10:56:37.016518 12481 heartbeater.cc:511] Master 127.30.218.62:37811 requested a full tablet report, sending...
I20260806 10:56:37.018340 12481 heartbeater.cc:464] Registering TS with master...
I20260806 10:56:37.018807 12481 heartbeater.cc:511] Master 127.30.218.62:37811 requested a full tablet report, sending...
W20260806 10:56:37.020259 12481 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 10:56:37.388278 12485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:56:37.388677 12485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:56:37.388830 12485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:56:37.397128 12485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:56:37.397307 12485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:56:37.397512 12485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:56:37.409538 12485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:42659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.218.20:50737/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:34293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=38931
--tserver_master_addrs=127.30.218.62:37811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:56:37.411094 12485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:56:37.412575 12485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:56:37.429301 12501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:37.429354 12500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:37.429929 12485 server_base.cc:1034] running on GCE node
W20260806 10:56:37.430135 12503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:37.430876 12485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:56:37.431958 12485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:56:37.433187 12485 hybrid_clock.cc:648] HybridClock initialized: now 1786013797433069 us; error 118 us; skew 500 ppm
I20260806 10:56:37.433593 12485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:56:37 dist-test-slave-49h3 krb5kdc[11250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013797, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:37.437520 12485 init.cc:377] Logged in from keytab as kudu/127.30.218.2@KRBTEST.COM (short username kudu)
I20260806 10:56:37.446743 12485 webserver.cc:460] Webserver started at http://127.30.218.2:38931/ using document root <none> and password file <none>
I20260806 10:56:37.448057 12485 fs_manager.cc:359] Metadata directory not provided
I20260806 10:56:37.448342 12485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 10:56:37 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013797, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for HTTP/127.30.218.20@KRBTEST.COM
I20260806 10:56:37.519697 12485 fs_manager.cc:709] Time spent opening directory manager: real 0.025s user 0.004s sys 0.016s
I20260806 10:56:37.529718 12513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:37.531795 12485 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.005s
I20260806 10:56:37.532058 12485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "3ff86ea049094be6ba7d367280fb3693"
format_stamp: "Formatted at 2026-08-06 10:56:34 on dist-test-slave-49h3"
server_key: "0a2a8309151f5954411b64b0cae101db"
server_key_iv: "d4b5cdb79306f81e22f978ac8d940776"
server_key_version: "kuduclusterkey@0"
I20260806 10:56:37.533656 12485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:56:37.572813 12485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:56:37.578599 12485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:56:37.579063 12485 kserver.cc:163] Server-wide thread pool size limit: 3276
Aug 06 10:56:37 dist-test-slave-49h3 krb5kdc[11250](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786013797, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:37.586733 12485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:56:37.595350 12485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:56:37.595584 12485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 10:56:37.595765 12485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:56:37.595897 12485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:37.713738 12627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:34293 every 8 connection(s)
I20260806 10:56:37.714289 12485 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:34293
I20260806 10:56:37.716545 12485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:56:37.724977 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 12485
I20260806 10:56:37.725797 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.218.20:50737/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:44267
--local_ip_for_outbound_sockets=127.30.218.3
--tserver_master_addrs=127.30.218.62:37811
--webserver_port=40993
--webserver_interface=127.30.218.3
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:42659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.218.20:50737/kms with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 10:56:37 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013797, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:56:37.774402 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:37.729133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:59021 (local address 127.30.218.62:37811)
0806 10:56:37.751743 (+ 22610us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:37.751749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:37.751777 (+ 28us) server_negotiation.cc:407] Connection header received
0806 10:56:37.751853 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:37.751858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:37.751949 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:37.752086 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:37.755063 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:37.756178 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:37.758751 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:37.759338 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:37.759461 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:37.764987 (+ 5526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:37.765000 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:37.765005 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:37.765081 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:37.769523 (+ 4442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:37.770868 (+ 1345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:37.770876 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:37.770880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:37.771080 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:37.771778 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:37.771784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:37.771788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:37.772233 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:37.772414 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:37.773695 (+ 1281us) server_negotiation.cc:299] Negotiation successful
0806 10:56:37.773983 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22450,"thread_start_us":23058,"threads_started":1}
I20260806 10:56:37.776146 12631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:37.729743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:37811 (local address 127.30.218.2:59021)
0806 10:56:37.744072 (+ 14329us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:37.744138 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:37.746639 (+ 2501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:37.752925 (+ 6286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:37.752940 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:37.753717 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:56:37.754789 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:37.754820 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:37.756407 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:37.756412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:37.758519 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:37.758539 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:37.759836 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:37.759845 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:37.760183 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:37.761020 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:56:37.761053 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:56:37.764240 (+ 3187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:56:37.769794 (+ 5554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:37.769807 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:37.769829 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:37.770649 (+ 820us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:37.771237 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:37.771243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:37.771247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:37.771612 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:37.772513 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:56:37.772523 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:56:37.773075 (+ 552us) client_negotiation.cc:769] Sending connection context
0806 10:56:37.773513 (+ 438us) client_negotiation.cc:240] Negotiation successful
0806 10:56:37.774203 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13914,"thread_start_us":17225,"threads_started":1}
I20260806 10:56:37.778371 12628 heartbeater.cc:347] Connected to a master server at 127.30.218.62:37811
I20260806 10:56:37.778725 12628 heartbeater.cc:464] Registering TS with master...
I20260806 10:56:37.779656 12628 heartbeater.cc:511] Master 127.30.218.62:37811 requested a full tablet report, sending...
I20260806 10:56:37.782204 12282 ts_manager.cc:194] Registered new tserver with Master: 3ff86ea049094be6ba7d367280fb3693 (127.30.218.2:34293)
I20260806 10:56:37.784686 12282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.2@KRBTEST.COM'} at 127.30.218.2:59021
I20260806 10:56:38.020432 12481 heartbeater.cc:464] Registering TS with master...
I20260806 10:56:38.020978 12481 heartbeater.cc:511] Master 127.30.218.62:37811 requested a full tablet report, sending...
I20260806 10:56:38.022647 12282 ts_manager.cc:194] Registered new tserver with Master: 83af05218d3d419bb47d69bf7944d578 (127.30.218.1:45481)
I20260806 10:56:38.024187 12282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.1@KRBTEST.COM'} at 127.30.218.1:56093
Aug 06 10:56:38 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013797, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for HTTP/127.30.218.20@KRBTEST.COM
W20260806 10:56:38.241703 12632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:56:38.242233 12632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:56:38.242430 12632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:56:38.254504 12632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:56:38.254760 12632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:56:38.254989 12632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:56:38.271492 12632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:42659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.218.20:50737/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:44267
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=40993
--tserver_master_addrs=127.30.218.62:37811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:56:38.273578 12632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:56:38.275348 12632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:56:38.313977 12639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:38.318849 12642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:38.319398 12638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:38.319357 12632 server_base.cc:1034] running on GCE node
I20260806 10:56:38.320470 12632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:56:38.327291 12632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:56:38.330858 12632 hybrid_clock.cc:648] HybridClock initialized: now 1786013798330752 us; error 120 us; skew 500 ppm
I20260806 10:56:38.331445 12632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:56:38 dist-test-slave-49h3 krb5kdc[11250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013798, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:38.336144 12632 init.cc:377] Logged in from keytab as kudu/127.30.218.3@KRBTEST.COM (short username kudu)
I20260806 10:56:38.357949 12632 webserver.cc:460] Webserver started at http://127.30.218.3:40993/ using document root <none> and password file <none>
I20260806 10:56:38.359198 12632 fs_manager.cc:359] Metadata directory not provided
I20260806 10:56:38.359524 12632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 10:56:38 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013798, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for HTTP/127.30.218.20@KRBTEST.COM
I20260806 10:56:38.397683 12632 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.006s sys 0.004s
I20260806 10:56:38.406503 12649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:38.408883 12632 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.005s sys 0.003s
I20260806 10:56:38.409235 12632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "762824a02e294c1897162b92e75fa8ea"
format_stamp: "Formatted at 2026-08-06 10:56:35 on dist-test-slave-49h3"
server_key: "a20d887ca29e851563cadd82a59ab3c0"
server_key_iv: "68456fea9d731bca66de8a1d35a1315e"
server_key_version: "kuduclusterkey@0"
I20260806 10:56:38.410938 12632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:56:38.473107 12632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:56:38.479080 12632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:56:38.479492 12632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:56:38.487171 12632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:56:38.495556 12632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:56:38.495772 12632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 10:56:38.495942 12632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:56:38.496169 12632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 10:56:38 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013797, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for HTTP/127.30.218.20@KRBTEST.COM
I20260806 10:56:38.790505 12628 heartbeater.cc:503] Master 127.30.218.62:37811 was elected leader, sending a full tablet report...
I20260806 10:56:39.009734 12632 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:44267
I20260806 10:56:39.009815 12770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:44267 every 8 connection(s)
I20260806 10:56:39.011309 12632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:56:39.017235 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 12632
I20260806 10:56:39.028182 12481 heartbeater.cc:503] Master 127.30.218.62:37811 was elected leader, sending a full tablet report...
Aug 06 10:56:39 dist-test-slave-49h3 krb5kdc[11250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013798, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:56:39.043315 12773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.014469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:44917 (local address 127.30.218.62:37811)
0806 10:56:39.020501 (+ 6032us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:39.020507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:39.020544 (+ 37us) server_negotiation.cc:407] Connection header received
0806 10:56:39.023309 (+ 2765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:39.023317 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:39.023475 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:39.023584 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:39.026944 (+ 3360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.028295 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.030990 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.031697 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.031837 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:39.035910 (+ 4073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:39.035917 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:39.035921 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:39.035991 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:39.039609 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:39.040570 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:39.040576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:39.040580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:39.040724 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:39.041230 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:39.041236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:39.041240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:39.041653 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:39.041827 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:39.042632 (+ 805us) server_negotiation.cc:299] Negotiation successful
0806 10:56:39.042915 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5903,"thread_start_us":6596,"threads_started":1}
I20260806 10:56:39.044301 12774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.015375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:37811 (local address 127.30.218.3:44917)
0806 10:56:39.019853 (+ 4478us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:39.019904 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:39.022711 (+ 2807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:39.024189 (+ 1478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:39.024205 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:39.025547 (+ 1342us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:56:39.026630 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:39.026662 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.028609 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.028613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:39.030791 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:39.030812 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.032009 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.032019 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:39.032294 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:39.032977 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:56:39.033008 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:56:39.035637 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:56:39.039777 (+ 4140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:39.039787 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:39.039804 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:39.040378 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:39.040850 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:39.040854 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:39.040857 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:39.041099 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:39.041814 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:56:39.041822 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:56:39.042221 (+ 399us) client_negotiation.cc:769] Sending connection context
0806 10:56:39.042489 (+ 268us) client_negotiation.cc:240] Negotiation successful
0806 10:56:39.043097 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4125,"thread_start_us":2581,"threads_started":1}
I20260806 10:56:39.046078 12771 heartbeater.cc:347] Connected to a master server at 127.30.218.62:37811
I20260806 10:56:39.046444 12771 heartbeater.cc:464] Registering TS with master...
I20260806 10:56:39.047334 12771 heartbeater.cc:511] Master 127.30.218.62:37811 requested a full tablet report, sending...
I20260806 10:56:39.048472 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.018470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32890 (local address 127.30.218.62:37811)
0806 10:56:39.029252 (+ 10782us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:39.029258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:39.029843 (+ 585us) server_negotiation.cc:407] Connection header received
0806 10:56:39.029904 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:39.029909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:39.030014 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:39.030109 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:39.032142 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.033258 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.035080 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.035702 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.035828 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:39.039393 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:39.039469 (+ 76us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:39.039475 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:39.039534 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:39.042915 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:39.046029 (+ 3114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:39.046035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:39.046039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:39.046227 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:39.046752 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:39.046758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:39.046761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:39.047132 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:39.047437 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:39.047781 (+ 344us) server_negotiation.cc:299] Negotiation successful
0806 10:56:39.048034 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10652,"spinlock_wait_cycles":9856,"thread_start_us":10494,"threads_started":1}
I20260806 10:56:39.049147 12282 ts_manager.cc:194] Registered new tserver with Master: 762824a02e294c1897162b92e75fa8ea (127.30.218.3:44267)
I20260806 10:56:39.050102 12282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.3@KRBTEST.COM'} at 127.30.218.3:44917
I20260806 10:56:39.051671 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:56:39.070143 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.054923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32904 (local address 127.30.218.62:37811)
0806 10:56:39.055134 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:39.055139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:39.055797 (+ 658us) server_negotiation.cc:407] Connection header received
0806 10:56:39.055987 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:39.055992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:39.056074 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:39.056177 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:39.057856 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.058939 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.060364 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.061010 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.061141 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:39.063933 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:39.063940 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:39.063944 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:39.063995 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:39.067329 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:39.068137 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:39.068142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:39.068146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:39.068330 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:39.068666 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:39.068670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:39.068673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:39.069023 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:39.069283 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:39.069587 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 10:56:39.069826 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 10:56:39.075659 12281 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:56:39.083478 12281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:56:39.113981 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.099283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:34293 (local address 127.0.0.1:45384)
0806 10:56:39.103670 (+ 4387us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:39.103726 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:39.103960 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:39.105179 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:39.105186 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:39.105218 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:56:39.105799 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:39.105814 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.107905 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.107910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:39.110476 (+ 2566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:39.110495 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.112868 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.112875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:39.113106 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:39.113163 (+ 57us) client_negotiation.cc:769] Sending connection context
0806 10:56:39.113313 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 10:56:39.113565 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2603,"thread_start_us":2421,"threads_started":1}
I20260806 10:56:39.115155 12787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.099279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45384 (local address 127.30.218.2:34293)
0806 10:56:39.104617 (+ 5338us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:39.104624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:39.104651 (+ 27us) server_negotiation.cc:407] Connection header received
0806 10:56:39.104743 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:39.104750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:39.104955 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:39.105152 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:56:39.106056 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.107671 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.110687 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.112274 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.112388 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:39.112639 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:39.114610 (+ 1971us) server_negotiation.cc:299] Negotiation successful
0806 10:56:39.114823 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5173,"thread_start_us":5006,"threads_started":1}
I20260806 10:56:39.117666 12790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.103988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:44267 (local address 127.0.0.1:56604)
0806 10:56:39.105889 (+ 1901us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:39.105907 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:39.106010 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:39.108862 (+ 2852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:39.108867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:39.108898 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:56:39.109288 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:39.109303 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.112723 (+ 3420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.112729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:39.114813 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:39.114828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.116829 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.116836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:39.117023 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:39.117059 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 10:56:39.117125 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 10:56:39.117280 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1763,"thread_start_us":1598,"threads_started":1}
I20260806 10:56:39.119028 12789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.103038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56604 (local address 127.30.218.3:44267)
0806 10:56:39.107878 (+ 4840us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:39.107887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:39.107921 (+ 34us) server_negotiation.cc:407] Connection header received
0806 10:56:39.108028 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:39.108036 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:39.108331 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:39.108538 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:56:39.109577 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.112559 (+ 2982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.115053 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.116540 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.118123 (+ 1583us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:39.118408 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:39.118494 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 10:56:39.118666 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4644,"thread_start_us":4498,"threads_started":1}
I20260806 10:56:39.123648 12699 tablet_service.cc:1467] Processing CreateTablet for tablet e828f45e86fc4236a3b8931491e8f23a (DEFAULT_TABLE table=test-table [id=729c0a657c284c089f61d3ede59241c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:56:39.124092 12562 tablet_service.cc:1467] Processing CreateTablet for tablet e828f45e86fc4236a3b8931491e8f23a (DEFAULT_TABLE table=test-table [id=729c0a657c284c089f61d3ede59241c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:56:39.125798 12562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e828f45e86fc4236a3b8931491e8f23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:56:39.125799 12699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e828f45e86fc4236a3b8931491e8f23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:56:39.127897 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.109196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:45481 (local address 127.0.0.1:54858)
0806 10:56:39.114267 (+ 5071us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:39.114289 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:39.114405 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:39.119071 (+ 4666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:39.119076 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:39.119108 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:56:39.119291 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:39.119302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.121532 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.121538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:39.124886 (+ 3348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:39.124907 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.127050 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.127057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:39.127272 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:39.127317 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 10:56:39.127402 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 10:56:39.127573 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4958,"thread_start_us":8465,"threads_started":1}
I20260806 10:56:39.129076 12791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.106323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54858 (local address 127.30.218.1:45481)
0806 10:56:39.118339 (+ 12016us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:39.118348 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:39.118385 (+ 37us) server_negotiation.cc:407] Connection header received
0806 10:56:39.118499 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:39.118508 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:39.118821 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:39.119484 (+ 663us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:56:39.119789 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.121314 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.125156 (+ 3842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.126822 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.128197 (+ 1375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:39.128463 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:39.128548 (+ 85us) server_negotiation.cc:299] Negotiation successful
0806 10:56:39.128747 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11807,"thread_start_us":11685,"threads_started":1}
I20260806 10:56:39.135417 12415 tablet_service.cc:1467] Processing CreateTablet for tablet e828f45e86fc4236a3b8931491e8f23a (DEFAULT_TABLE table=test-table [id=729c0a657c284c089f61d3ede59241c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:56:39.137207 12415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e828f45e86fc4236a3b8931491e8f23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:56:39.151741 12793 tablet_bootstrap.cc:492] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea: Bootstrap starting.
I20260806 10:56:39.154436 12794 tablet_bootstrap.cc:492] T e828f45e86fc4236a3b8931491e8f23a P 3ff86ea049094be6ba7d367280fb3693: Bootstrap starting.
I20260806 10:56:39.158311 12794 tablet_bootstrap.cc:654] T e828f45e86fc4236a3b8931491e8f23a P 3ff86ea049094be6ba7d367280fb3693: Neither blocks nor log segments found. Creating new log.
I20260806 10:56:39.158311 12793 tablet_bootstrap.cc:654] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea: Neither blocks nor log segments found. Creating new log.
I20260806 10:56:39.159884 12793 log.cc:824] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea: Log is configured to *not* fsync() on all Append() calls
I20260806 10:56:39.160200 12794 log.cc:824] T e828f45e86fc4236a3b8931491e8f23a P 3ff86ea049094be6ba7d367280fb3693: Log is configured to *not* fsync() on all Append() calls
I20260806 10:56:39.164091 12795 tablet_bootstrap.cc:492] T e828f45e86fc4236a3b8931491e8f23a P 83af05218d3d419bb47d69bf7944d578: Bootstrap starting.
I20260806 10:56:39.170904 12795 tablet_bootstrap.cc:654] T e828f45e86fc4236a3b8931491e8f23a P 83af05218d3d419bb47d69bf7944d578: Neither blocks nor log segments found. Creating new log.
I20260806 10:56:39.172632 12795 log.cc:824] T e828f45e86fc4236a3b8931491e8f23a P 83af05218d3d419bb47d69bf7944d578: Log is configured to *not* fsync() on all Append() calls
I20260806 10:56:39.172931 12794 tablet_bootstrap.cc:492] T e828f45e86fc4236a3b8931491e8f23a P 3ff86ea049094be6ba7d367280fb3693: No bootstrap required, opened a new log
I20260806 10:56:39.172932 12793 tablet_bootstrap.cc:492] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea: No bootstrap required, opened a new log
I20260806 10:56:39.173281 12794 ts_tablet_manager.cc:1397] T e828f45e86fc4236a3b8931491e8f23a P 3ff86ea049094be6ba7d367280fb3693: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.014s
I20260806 10:56:39.173290 12793 ts_tablet_manager.cc:1397] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.011s
I20260806 10:56:39.184808 12793 raft_consensus.cc:348] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ff86ea049094be6ba7d367280fb3693" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 34293 } } peers { permanent_uuid: "762824a02e294c1897162b92e75fa8ea" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 44267 } } peers { permanent_uuid: "83af05218d3d419bb47d69bf7944d578" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 45481 } }
I20260806 10:56:39.185493 12793 raft_consensus.cc:374] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:56:39.185655 12793 raft_consensus.cc:729] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 762824a02e294c1897162b92e75fa8ea, State: Initialized, Role: FOLLOWER
I20260806 10:56:39.186563 12793 consensus_queue.cc:260] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ff86ea049094be6ba7d367280fb3693" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 34293 } } peers { permanent_uuid: "762824a02e294c1897162b92e75fa8ea" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 44267 } } peers { permanent_uuid: "83af05218d3d419bb47d69bf7944d578" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 45481 } }
I20260806 10:56:39.187337 12794 raft_consensus.cc:348] T e828f45e86fc4236a3b8931491e8f23a P 3ff86ea049094be6ba7d367280fb3693 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ff86ea049094be6ba7d367280fb3693" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 34293 } } peers { permanent_uuid: "762824a02e294c1897162b92e75fa8ea" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 44267 } } peers { permanent_uuid: "83af05218d3d419bb47d69bf7944d578" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 45481 } }
I20260806 10:56:39.187947 12794 raft_consensus.cc:374] T e828f45e86fc4236a3b8931491e8f23a P 3ff86ea049094be6ba7d367280fb3693 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:56:39.188092 12794 raft_consensus.cc:729] T e828f45e86fc4236a3b8931491e8f23a P 3ff86ea049094be6ba7d367280fb3693 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ff86ea049094be6ba7d367280fb3693, State: Initialized, Role: FOLLOWER
I20260806 10:56:39.188968 12794 consensus_queue.cc:260] T e828f45e86fc4236a3b8931491e8f23a P 3ff86ea049094be6ba7d367280fb3693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ff86ea049094be6ba7d367280fb3693" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 34293 } } peers { permanent_uuid: "762824a02e294c1897162b92e75fa8ea" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 44267 } } peers { permanent_uuid: "83af05218d3d419bb47d69bf7944d578" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 45481 } }
I20260806 10:56:39.191349 12794 ts_tablet_manager.cc:1428] T e828f45e86fc4236a3b8931491e8f23a P 3ff86ea049094be6ba7d367280fb3693: Time spent starting tablet: real 0.018s user 0.014s sys 0.000s
I20260806 10:56:39.194228 12795 tablet_bootstrap.cc:492] T e828f45e86fc4236a3b8931491e8f23a P 83af05218d3d419bb47d69bf7944d578: No bootstrap required, opened a new log
I20260806 10:56:39.194327 12771 heartbeater.cc:503] Master 127.30.218.62:37811 was elected leader, sending a full tablet report...
I20260806 10:56:39.194588 12795 ts_tablet_manager.cc:1397] T e828f45e86fc4236a3b8931491e8f23a P 83af05218d3d419bb47d69bf7944d578: Time spent bootstrapping tablet: real 0.031s user 0.016s sys 0.009s
I20260806 10:56:39.195039 12793 ts_tablet_manager.cc:1428] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea: Time spent starting tablet: real 0.022s user 0.008s sys 0.015s
I20260806 10:56:39.206259 12795 raft_consensus.cc:348] T e828f45e86fc4236a3b8931491e8f23a P 83af05218d3d419bb47d69bf7944d578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ff86ea049094be6ba7d367280fb3693" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 34293 } } peers { permanent_uuid: "762824a02e294c1897162b92e75fa8ea" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 44267 } } peers { permanent_uuid: "83af05218d3d419bb47d69bf7944d578" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 45481 } }
I20260806 10:56:39.206866 12795 raft_consensus.cc:374] T e828f45e86fc4236a3b8931491e8f23a P 83af05218d3d419bb47d69bf7944d578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:56:39.206985 12795 raft_consensus.cc:729] T e828f45e86fc4236a3b8931491e8f23a P 83af05218d3d419bb47d69bf7944d578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83af05218d3d419bb47d69bf7944d578, State: Initialized, Role: FOLLOWER
I20260806 10:56:39.207413 12799 raft_consensus.cc:482] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:56:39.207723 12799 raft_consensus.cc:504] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ff86ea049094be6ba7d367280fb3693" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 34293 } } peers { permanent_uuid: "762824a02e294c1897162b92e75fa8ea" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 44267 } } peers { permanent_uuid: "83af05218d3d419bb47d69bf7944d578" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 45481 } }
I20260806 10:56:39.207773 12795 consensus_queue.cc:260] T e828f45e86fc4236a3b8931491e8f23a P 83af05218d3d419bb47d69bf7944d578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ff86ea049094be6ba7d367280fb3693" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 34293 } } peers { permanent_uuid: "762824a02e294c1897162b92e75fa8ea" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 44267 } } peers { permanent_uuid: "83af05218d3d419bb47d69bf7944d578" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 45481 } }
I20260806 10:56:39.209805 12799 leader_election.cc:288] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ff86ea049094be6ba7d367280fb3693 (127.30.218.2:34293), 83af05218d3d419bb47d69bf7944d578 (127.30.218.1:45481)
I20260806 10:56:39.209939 12795 ts_tablet_manager.cc:1428] T e828f45e86fc4236a3b8931491e8f23a P 83af05218d3d419bb47d69bf7944d578: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260806 10:56:39.217540 12774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.210487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:34293 (local address 127.30.218.3:41809)
0806 10:56:39.210721 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:39.210741 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:39.210859 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:39.211373 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:39.211378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:39.211410 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:56:39.211621 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:39.211633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.213232 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.213238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:39.215225 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:39.215244 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.216681 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.216687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:39.216920 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:39.216956 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 10:56:39.217058 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 10:56:39.217217 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":6}
I20260806 10:56:39.218212 12787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.210797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:41809 (local address 127.30.218.2:34293)
0806 10:56:39.211006 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:39.211033 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:39.211055 (+ 22us) server_negotiation.cc:407] Connection header received
0806 10:56:39.211144 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:39.211149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:39.211239 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:39.211337 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:56:39.211886 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.213050 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.215441 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.216509 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.216678 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:39.216761 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:39.217809 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0806 10:56:39.217966 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 10:56:39.219331 12582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e828f45e86fc4236a3b8931491e8f23a" candidate_uuid: "762824a02e294c1897162b92e75fa8ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ff86ea049094be6ba7d367280fb3693" is_pre_election: true
I20260806 10:56:39.219970 12582 raft_consensus.cc:2454] T e828f45e86fc4236a3b8931491e8f23a P 3ff86ea049094be6ba7d367280fb3693 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 762824a02e294c1897162b92e75fa8ea in term 0.
I20260806 10:56:39.220242 12791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.211452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:52885 (local address 127.30.218.1:45481)
0806 10:56:39.211659 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:39.211664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:39.213172 (+ 1508us) server_negotiation.cc:407] Connection header received
0806 10:56:39.213487 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:39.213493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:39.213573 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:39.213635 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:56:39.214570 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.215679 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.218154 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.219249 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.219414 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:39.219493 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:39.219830 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 10:56:39.219970 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61,"spinlock_wait_cycles":27392}
I20260806 10:56:39.220275 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.210907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:45481 (local address 127.30.218.3:52885)
0806 10:56:39.213063 (+ 2156us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:39.213087 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:39.213306 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:39.213908 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:39.213912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:39.213940 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:56:39.214381 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:39.214395 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.215874 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.215879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:39.217979 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:39.217995 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.219452 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.219459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:39.219637 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:39.219670 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 10:56:39.219796 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 10:56:39.219939 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2030,"thread_start_us":1868,"threads_started":1}
I20260806 10:56:39.220774 12651 leader_election.cc:302] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ff86ea049094be6ba7d367280fb3693, 762824a02e294c1897162b92e75fa8ea; no voters:
I20260806 10:56:39.221169 12799 raft_consensus.cc:2781] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:56:39.221328 12799 raft_consensus.cc:482] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:56:39.221433 12799 raft_consensus.cc:3037] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:56:39.222121 12435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e828f45e86fc4236a3b8931491e8f23a" candidate_uuid: "762824a02e294c1897162b92e75fa8ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83af05218d3d419bb47d69bf7944d578" is_pre_election: true
I20260806 10:56:39.222708 12435 raft_consensus.cc:2454] T e828f45e86fc4236a3b8931491e8f23a P 83af05218d3d419bb47d69bf7944d578 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 762824a02e294c1897162b92e75fa8ea in term 0.
I20260806 10:56:39.224519 12799 raft_consensus.cc:504] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ff86ea049094be6ba7d367280fb3693" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 34293 } } peers { permanent_uuid: "762824a02e294c1897162b92e75fa8ea" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 44267 } } peers { permanent_uuid: "83af05218d3d419bb47d69bf7944d578" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 45481 } }
I20260806 10:56:39.225310 12799 leader_election.cc:288] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea [CANDIDATE]: Term 1 election: Requested vote from peers 3ff86ea049094be6ba7d367280fb3693 (127.30.218.2:34293), 83af05218d3d419bb47d69bf7944d578 (127.30.218.1:45481)
I20260806 10:56:39.226318 12582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e828f45e86fc4236a3b8931491e8f23a" candidate_uuid: "762824a02e294c1897162b92e75fa8ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ff86ea049094be6ba7d367280fb3693"
I20260806 10:56:39.226569 12582 raft_consensus.cc:3037] T e828f45e86fc4236a3b8931491e8f23a P 3ff86ea049094be6ba7d367280fb3693 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:56:39.226706 12435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e828f45e86fc4236a3b8931491e8f23a" candidate_uuid: "762824a02e294c1897162b92e75fa8ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83af05218d3d419bb47d69bf7944d578"
I20260806 10:56:39.226958 12435 raft_consensus.cc:3037] T e828f45e86fc4236a3b8931491e8f23a P 83af05218d3d419bb47d69bf7944d578 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:56:39.229228 12435 raft_consensus.cc:2454] T e828f45e86fc4236a3b8931491e8f23a P 83af05218d3d419bb47d69bf7944d578 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 762824a02e294c1897162b92e75fa8ea in term 1.
W20260806 10:56:39.229478 12482 tablet.cc:2369] T e828f45e86fc4236a3b8931491e8f23a P 83af05218d3d419bb47d69bf7944d578: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:56:39.230114 12582 raft_consensus.cc:2454] T e828f45e86fc4236a3b8931491e8f23a P 3ff86ea049094be6ba7d367280fb3693 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 762824a02e294c1897162b92e75fa8ea in term 1.
I20260806 10:56:39.230430 12651 leader_election.cc:302] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 762824a02e294c1897162b92e75fa8ea, 83af05218d3d419bb47d69bf7944d578; no voters:
I20260806 10:56:39.230784 12799 raft_consensus.cc:2781] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:56:39.231313 12799 raft_consensus.cc:686] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea [term 1 LEADER]: Becoming Leader. State: Replica: 762824a02e294c1897162b92e75fa8ea, State: Running, Role: LEADER
I20260806 10:56:39.231835 12799 consensus_queue.cc:237] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ff86ea049094be6ba7d367280fb3693" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 34293 } } peers { permanent_uuid: "762824a02e294c1897162b92e75fa8ea" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 44267 } } peers { permanent_uuid: "83af05218d3d419bb47d69bf7944d578" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 45481 } }
W20260806 10:56:39.235487 12629 tablet.cc:2369] T e828f45e86fc4236a3b8931491e8f23a P 3ff86ea049094be6ba7d367280fb3693: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:56:39.237172 12282 catalog_manager.cc:5495] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 762824a02e294c1897162b92e75fa8ea (127.30.218.3). New cstate: current_term: 1 leader_uuid: "762824a02e294c1897162b92e75fa8ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ff86ea049094be6ba7d367280fb3693" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 34293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "762824a02e294c1897162b92e75fa8ea" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 44267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83af05218d3d419bb47d69bf7944d578" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 45481 } health_report { overall_health: UNKNOWN } } }
I20260806 10:56:39.269940 12789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.264060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56608 (local address 127.30.218.3:44267)
0806 10:56:39.264272 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:39.264277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:39.264413 (+ 136us) server_negotiation.cc:407] Connection header received
0806 10:56:39.264643 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:39.264647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:39.264729 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:39.264866 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:56:39.265211 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.266128 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.267540 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.267958 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.268087 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:39.268669 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:56:39.269344 (+ 675us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:56:39.269515 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:39.269627 (+ 112us) server_negotiation.cc:299] Negotiation successful
0806 10:56:39.269712 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260806 10:56:39.270377 12772 tablet.cc:2369] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:56:39.278978 12582 raft_consensus.cc:1261] T e828f45e86fc4236a3b8931491e8f23a P 3ff86ea049094be6ba7d367280fb3693 [term 1 FOLLOWER]: Refusing update from remote peer 762824a02e294c1897162b92e75fa8ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:56:39.278982 12435 raft_consensus.cc:1261] T e828f45e86fc4236a3b8931491e8f23a P 83af05218d3d419bb47d69bf7944d578 [term 1 FOLLOWER]: Refusing update from remote peer 762824a02e294c1897162b92e75fa8ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:56:39.280112 12799 consensus_queue.cc:1035] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ff86ea049094be6ba7d367280fb3693" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:56:39.280436 12803 consensus_queue.cc:1035] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "83af05218d3d419bb47d69bf7944d578" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 45481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:56:39.292843 12807 mvcc.cc:205] Tried to move back new op lower bound from 7315512521834733568 to 7315512521657733120. Current Snapshot: MvccSnapshot[applied={T|T < 7315512521834733568}]
I20260806 10:56:39.297034 12809 mvcc.cc:205] Tried to move back new op lower bound from 7315512521834733568 to 7315512521657733120. Current Snapshot: MvccSnapshot[applied={T|T < 7315512521834733568}]
I20260806 10:56:39.314036 12282 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:56:39.314373 12282 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:56:39.319162 12282 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d10df381f8c40279a3d7696621dabbd: Sending DeleteTablet for 3 replicas of tablet e828f45e86fc4236a3b8931491e8f23a
I20260806 10:56:39.320767 12699 tablet_service.cc:1514] Processing DeleteTablet for tablet e828f45e86fc4236a3b8931491e8f23a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:56:39 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:56604
I20260806 10:56:39.320880 12415 tablet_service.cc:1514] Processing DeleteTablet for tablet e828f45e86fc4236a3b8931491e8f23a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:56:39 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:54858
I20260806 10:56:39.321022 12562 tablet_service.cc:1514] Processing DeleteTablet for tablet e828f45e86fc4236a3b8931491e8f23a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:56:39 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:45384
I20260806 10:56:39.326457 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 12319
I20260806 10:56:39.343111 12818 tablet_replica.cc:330] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea: stopping tablet replica
I20260806 10:56:39.343791 12819 tablet_replica.cc:330] T e828f45e86fc4236a3b8931491e8f23a P 83af05218d3d419bb47d69bf7944d578: stopping tablet replica
I20260806 10:56:39.344470 12819 raft_consensus.cc:2229] T e828f45e86fc4236a3b8931491e8f23a P 83af05218d3d419bb47d69bf7944d578 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:56:39.345082 12818 raft_consensus.cc:2229] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:56:39.345769 12819 raft_consensus.cc:2258] T e828f45e86fc4236a3b8931491e8f23a P 83af05218d3d419bb47d69bf7944d578 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:56:39.346695 12818 raft_consensus.cc:2258] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:56:39.348183 12820 tablet_replica.cc:330] T e828f45e86fc4236a3b8931491e8f23a P 3ff86ea049094be6ba7d367280fb3693: stopping tablet replica
I20260806 10:56:39.348632 12820 raft_consensus.cc:2229] T e828f45e86fc4236a3b8931491e8f23a P 3ff86ea049094be6ba7d367280fb3693 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:56:39.349190 12820 raft_consensus.cc:2258] T e828f45e86fc4236a3b8931491e8f23a P 3ff86ea049094be6ba7d367280fb3693 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:56:39.350072 12819 ts_tablet_manager.cc:1905] T e828f45e86fc4236a3b8931491e8f23a P 83af05218d3d419bb47d69bf7944d578: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:56:39.351142 12818 ts_tablet_manager.cc:1905] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:56:39.357792 12820 ts_tablet_manager.cc:1905] T e828f45e86fc4236a3b8931491e8f23a P 3ff86ea049094be6ba7d367280fb3693: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:56:39.358333 12791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.327534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54862 (local address 127.30.218.1:45481)
0806 10:56:39.327720 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:39.327726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:39.329511 (+ 1785us) server_negotiation.cc:407] Connection header received
0806 10:56:39.329606 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:39.329612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:39.329702 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:39.329793 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:39.332004 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.333217 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.335415 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.336186 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.336329 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:39.340701 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:39.340728 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:39.340754 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:39.340823 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:39.345143 (+ 4320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:39.346301 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:39.346311 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:39.346314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:39.346498 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:39.347015 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:39.347020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:39.347023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:39.348201 (+ 1178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:39.348516 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:39.357594 (+ 9078us) server_negotiation.cc:299] Negotiation successful
0806 10:56:39.357910 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 10:56:39.359342 12476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:56:39.364593 12818 ts_tablet_manager.cc:1918] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:56:39.364809 12818 log.cc:1196] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/e828f45e86fc4236a3b8931491e8f23a
I20260806 10:56:39.365374 12818 ts_tablet_manager.cc:1939] T e828f45e86fc4236a3b8931491e8f23a P 762824a02e294c1897162b92e75fa8ea: Deleting consensus metadata
I20260806 10:56:39.367272 12820 ts_tablet_manager.cc:1918] T e828f45e86fc4236a3b8931491e8f23a P 3ff86ea049094be6ba7d367280fb3693: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:56:39.367446 12820 log.cc:1196] T e828f45e86fc4236a3b8931491e8f23a P 3ff86ea049094be6ba7d367280fb3693: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/e828f45e86fc4236a3b8931491e8f23a
I20260806 10:56:39.367971 12820 ts_tablet_manager.cc:1939] T e828f45e86fc4236a3b8931491e8f23a P 3ff86ea049094be6ba7d367280fb3693: Deleting consensus metadata
I20260806 10:56:39.370095 12266 catalog_manager.cc:4841] TS 3ff86ea049094be6ba7d367280fb3693 (127.30.218.2:34293): tablet e828f45e86fc4236a3b8931491e8f23a (table test-table [id=729c0a657c284c089f61d3ede59241c3]) successfully deleted
I20260806 10:56:39.371299 12266 catalog_manager.cc:4841] TS 762824a02e294c1897162b92e75fa8ea (127.30.218.3:44267): tablet e828f45e86fc4236a3b8931491e8f23a (table test-table [id=729c0a657c284c089f61d3ede59241c3]) successfully deleted
I20260806 10:56:39.453539 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 12319
I20260806 10:56:39.470220 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 12485
W20260806 10:56:39.470233 12266 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.218.1:45481 (error 108)
I20260806 10:56:39.472311 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.471692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:45481 (local address 127.0.0.1:54870)
0806 10:56:39.471979 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:39.472102 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:45481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260806 10:56:39.472560 12266 catalog_manager.cc:4584] TS 83af05218d3d419bb47d69bf7944d578 (127.30.218.1:45481): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e828f45e86fc4236a3b8931491e8f23a: Network error: Client connection negotiation failed: client connection to 127.30.218.1:45481: connect: Connection refused (error 111)
I20260806 10:56:39.484748 12787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.471048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45398 (local address 127.30.218.2:34293)
0806 10:56:39.471244 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:39.471250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:39.471495 (+ 245us) server_negotiation.cc:407] Connection header received
0806 10:56:39.471570 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:39.471573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:39.471647 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:39.471733 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:39.473164 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.473942 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.475205 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.475638 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.475754 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:39.478746 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:39.478762 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:39.478779 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:39.478816 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:39.481856 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:39.482621 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:39.482630 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:39.482632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:39.482768 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:39.483145 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:39.483149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:39.483151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:39.483530 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:39.483839 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:39.484152 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 10:56:39.484386 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 10:56:39.485572 12623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:56:39.526252 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.525765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:45481 (local address 127.0.0.1:54872)
0806 10:56:39.525983 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:39.526099 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:45481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 10:56:39.527396 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.527013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:45481 (local address 127.0.0.1:54884)
0806 10:56:39.527163 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:39.527249 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:45481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 10:56:39.588435 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.587833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:45481 (local address 127.0.0.1:54890)
0806 10:56:39.588120 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:39.588249 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:45481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260806 10:56:39.589838 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.589346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:45481 (local address 127.0.0.1:54894)
0806 10:56:39.589548 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:39.589668 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:45481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 10:56:39.606693 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 12485
I20260806 10:56:39.625018 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 12632
I20260806 10:56:39.643090 12789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.625996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56622 (local address 127.30.218.3:44267)
0806 10:56:39.626187 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:39.626192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:39.626212 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:56:39.626428 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:39.626432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:39.626519 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:39.626605 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:39.628459 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.629301 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.631514 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:39.632004 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:39.632114 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:39.635751 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:39.635767 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:39.635784 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:39.635831 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:39.639719 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:39.640593 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:39.640604 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:39.640608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:39.640831 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:39.641219 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:39.641224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:39.641227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:39.641768 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:39.642126 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:39.642410 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 10:56:39.642683 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 10:56:39.643859 12766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:56:39.670457 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.669949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:45481 (local address 127.0.0.1:54908)
0806 10:56:39.670199 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:39.670299 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:45481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 10:56:39.671568 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.671135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:45481 (local address 127.0.0.1:54912)
0806 10:56:39.671316 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:39.671396 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:45481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260806 10:56:39.769632 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 12632
I20260806 10:56:39.788683 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 12247
I20260806 10:56:39.790004 12312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:56:39.905416 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:39.904469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:45481 (local address 127.0.0.1:54924)
0806 10:56:39.905005 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:39.905161 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:45481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296}
I20260806 10:56:39.905655 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 12247
I20260806 10:56:39.931950 31592 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 10:56:40.280506 31592 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 10:56:40.280808 31592 mini_ranger.cc:66] Stopping Ranger...
I20260806 10:56:40.637094 31592 mini_ranger.cc:68] Stopped Ranger
2026-08-06 10:56:40.637 UTC [11280] LOG: received smart shutdown request
2026-08-06 10:56:40.640 UTC [11280] LOG: background worker "logical replication launcher" (PID 11288) exited with exit code 1
2026-08-06 10:56:40.640 UTC [11283] LOG: shutting down
2026-08-06 10:56:40.803 UTC [11280] LOG: database system is shut down
2026-08-06T10:56:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (56481 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:56:41 dist-test-slave-49h3 krb5kdc[12834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:56:41 dist-test-slave-49h3 krb5kdc[12834](info): set up 2 sockets
Aug 06 10:56:41 dist-test-slave-49h3 krb5kdc[12834](info): commencing operation
krb5kdc: starting...
W20260806 10:56:42.018764 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.004s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:56:42 dist-test-slave-49h3 krb5kdc[12834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:56:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:56:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:56:42.345088 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:41563
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:38323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:41563,127.30.218.61:33853,127.30.218.60:33733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:56:42.731124 12847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:56:42.731503 12847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:56:42.731642 12847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:56:42.739358 12847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:56:42.739486 12847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:56:42.739550 12847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:56:42.739645 12847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:56:42.739697 12847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:56:42.750247 12847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:38323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:41563,127.30.218.61:33853,127.30.218.60:33733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:41563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:56:42.751901 12847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:56:42.753494 12847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:56:42.766697 12855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:42.766651 12852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:42.766697 12853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:42.767648 12847 server_base.cc:1034] running on GCE node
I20260806 10:56:42.768535 12847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:56:42.770205 12847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:56:42.771463 12847 hybrid_clock.cc:648] HybridClock initialized: now 1786013802771394 us; error 38 us; skew 500 ppm
I20260806 10:56:42.771883 12847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:56:42 dist-test-slave-49h3 krb5kdc[12834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013802, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:42.778520 12847 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:56:42.780329 12847 webserver.cc:460] Webserver started at http://127.30.218.62:34357/ using document root <none> and password file <none>
I20260806 10:56:42.781163 12847 fs_manager.cc:359] Metadata directory not provided
I20260806 10:56:42.781287 12847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:56:42.781687 12847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:56:42.785539 12847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "69bf9380878147df8ee0b27298cd74c6"
format_stamp: "Formatted at 2026-08-06 10:56:42 on dist-test-slave-49h3"
server_key: "1376d98888893f85a1f9f376e0bd732d"
server_key_iv: "f0256a85dcf5782e73bdd703fc9cb0c7"
server_key_version: "encryptionkey@0"
I20260806 10:56:42.786536 12847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "69bf9380878147df8ee0b27298cd74c6"
format_stamp: "Formatted at 2026-08-06 10:56:42 on dist-test-slave-49h3"
server_key: "1376d98888893f85a1f9f376e0bd732d"
server_key_iv: "f0256a85dcf5782e73bdd703fc9cb0c7"
server_key_version: "encryptionkey@0"
I20260806 10:56:42.792555 12847 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 10:56:42.796336 12862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:42.797853 12847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 10:56:42.798029 12847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "69bf9380878147df8ee0b27298cd74c6"
format_stamp: "Formatted at 2026-08-06 10:56:42 on dist-test-slave-49h3"
server_key: "1376d98888893f85a1f9f376e0bd732d"
server_key_iv: "f0256a85dcf5782e73bdd703fc9cb0c7"
server_key_version: "encryptionkey@0"
I20260806 10:56:42.798225 12847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:56:42.829360 12847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:56:42.835083 12847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:56:42.835457 12847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:56:42.856220 12847 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:41563
I20260806 10:56:42.856244 12913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:41563 every 8 connection(s)
I20260806 10:56:42.858003 12847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:56:42.862746 12914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:56:42.862730 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 12847
I20260806 10:56:42.863016 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:56:42.863989 31592 external_mini_cluster.cc:1428] Setting key 395cf3a2a2a315af8bd3d95cca975907
I20260806 10:56:42.873085 12914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41563 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:56:42 dist-test-slave-49h3 krb5kdc[12834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013802, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.218.61@KRBTEST.COM; defaulting to no policy
I20260806 10:56:42.905114 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:42.877847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52852 (local address 127.30.218.62:41563)
0806 10:56:42.882805 (+ 4958us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:42.882821 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:42.883181 (+ 360us) server_negotiation.cc:407] Connection header received
0806 10:56:42.886092 (+ 2911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:42.886099 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:42.886392 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:42.886582 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:42.888399 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:42.889832 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:42.891908 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:42.892533 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:42.892680 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:42.896957 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:42.896965 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:42.896981 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:42.897012 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:42.900160 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:42.900887 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:42.900895 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:42.900897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:42.901052 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:42.901433 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:42.901437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:42.901439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:42.902078 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:42.902282 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:42.902854 (+ 572us) server_negotiation.cc:299] Negotiation successful
0806 10:56:42.903325 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":4646,"spinlock_wait_cycles":68992,"thread_start_us":4330,"threads_started":1}
I20260806 10:56:42.905126 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:42.877846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:41563 (local address 127.0.0.1:52852)
0806 10:56:42.882953 (+ 5107us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:42.883012 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:42.885613 (+ 2601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:42.886664 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:42.886675 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:42.887639 (+ 964us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:56:42.888174 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:42.888200 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:42.889996 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:42.890002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:42.891754 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:42.891770 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:42.892801 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:42.892809 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:42.893003 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:42.893823 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:56:42.893854 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:56:42.896724 (+ 2870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:56:42.900302 (+ 3578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:42.900314 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:42.900330 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:42.900755 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:42.901163 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:42.901168 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:42.901171 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:42.901331 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:42.902212 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:56:42.902221 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:56:42.902444 (+ 223us) client_negotiation.cc:769] Sending connection context
0806 10:56:42.902703 (+ 259us) client_negotiation.cc:240] Negotiation successful
0806 10:56:42.903439 (+ 736us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4753,"mutex_wait_us":220,"thread_start_us":4362,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.61@KRBTEST.COM" created.
I20260806 10:56:42.909037 12914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:56:42.910465 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:42.909862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33853 (local address 127.0.0.1:36144)
0806 10:56:42.910195 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:42.910324 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218}
W20260806 10:56:42.910742 12865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33853: connect: Connection refused (error 111)
I20260806 10:56:42.912634 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:42.912176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33853 (local address 127.0.0.1:36152)
0806 10:56:42.912368 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:42.912449 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260806 10:56:42.912850 12914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.61:33853: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33853: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.61.keytab.
Entry for principal kudu/127.30.218.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.61.keytab.
I20260806 10:56:42.947254 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.61:33853
--webserver_interface=127.30.218.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.61.keytab
--principal=kudu/127.30.218.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:38323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:41563,127.30.218.61:33853,127.30.218.60:33733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:56:42.962314 12914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } attempt: 1
I20260806 10:56:42.964128 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:42.963364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33853 (local address 127.0.0.1:36160)
0806 10:56:42.963646 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:42.963875 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260806 10:56:42.965886 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:42.965193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33853 (local address 127.0.0.1:36164)
0806 10:56:42.965429 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:42.965609 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260806 10:56:42.966293 12914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.61:33853: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33853: connect: Connection refused (error 111)
I20260806 10:56:43.034605 12914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } attempt: 2
I20260806 10:56:43.036067 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.035518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33853 (local address 127.0.0.1:36174)
0806 10:56:43.035765 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.035882 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260806 10:56:43.037175 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.036762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33853 (local address 127.0.0.1:36180)
0806 10:56:43.036919 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.037006 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 10:56:43.037351 12914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.61:33853: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33853: connect: Connection refused (error 111)
I20260806 10:56:43.128680 12914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } attempt: 3
I20260806 10:56:43.130285 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.129641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33853 (local address 127.0.0.1:36196)
0806 10:56:43.129923 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.130082 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260806 10:56:43.131490 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.131019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33853 (local address 127.0.0.1:36198)
0806 10:56:43.131197 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.131297 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 10:56:43.131770 12914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.61:33853: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33853: connect: Connection refused (error 111)
I20260806 10:56:43.275087 12914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } attempt: 4
I20260806 10:56:43.276569 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.275848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33853 (local address 127.0.0.1:36214)
0806 10:56:43.276243 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.276365 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200}
I20260806 10:56:43.277768 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.277273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33853 (local address 127.0.0.1:36224)
0806 10:56:43.277432 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.277601 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 10:56:43.278007 12914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.61:33853: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33853: connect: Connection refused (error 111)
W20260806 10:56:43.339445 12921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:56:43.339776 12921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:56:43.339869 12921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:56:43.347455 12921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:56:43.347596 12921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:56:43.347653 12921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:56:43.347711 12921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:56:43.347756 12921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:56:43.358253 12921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:38323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:41563,127.30.218.61:33853,127.30.218.60:33733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.61.keytab
--principal=kudu/127.30.218.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.61:33853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.218.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:56:43.359728 12921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:56:43.361161 12921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:56:43.373867 12929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:43.373808 12927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:43.374217 12926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:43.374666 12921 server_base.cc:1034] running on GCE node
I20260806 10:56:43.375586 12921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:56:43.377285 12921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:56:43.378531 12921 hybrid_clock.cc:648] HybridClock initialized: now 1786013803378477 us; error 80 us; skew 500 ppm
I20260806 10:56:43.378933 12921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:56:43 dist-test-slave-49h3 krb5kdc[12834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013803, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:43.382457 12921 init.cc:377] Logged in from keytab as kudu/127.30.218.61@KRBTEST.COM (short username kudu)
I20260806 10:56:43.384109 12921 webserver.cc:460] Webserver started at http://127.30.218.61:35141/ using document root <none> and password file <none>
I20260806 10:56:43.385003 12921 fs_manager.cc:359] Metadata directory not provided
I20260806 10:56:43.385147 12921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:56:43.385517 12921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:56:43.388996 12921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-1/data/instance:
uuid: "6b2d1625b6a949119fbdf233d7610ae8"
format_stamp: "Formatted at 2026-08-06 10:56:43 on dist-test-slave-49h3"
server_key: "0f2aee3401812d4ec196b1cc57e8a75b"
server_key_iv: "ac63ed697aab5be1c65c8753dc505139"
server_key_version: "encryptionkey@0"
I20260806 10:56:43.389987 12921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-1/wal/instance:
uuid: "6b2d1625b6a949119fbdf233d7610ae8"
format_stamp: "Formatted at 2026-08-06 10:56:43 on dist-test-slave-49h3"
server_key: "0f2aee3401812d4ec196b1cc57e8a75b"
server_key_iv: "ac63ed697aab5be1c65c8753dc505139"
server_key_version: "encryptionkey@0"
I20260806 10:56:43.396255 12921 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 10:56:43.400296 12936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:43.401993 12921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 10:56:43.402220 12921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-1/wal
uuid: "6b2d1625b6a949119fbdf233d7610ae8"
format_stamp: "Formatted at 2026-08-06 10:56:43 on dist-test-slave-49h3"
server_key: "0f2aee3401812d4ec196b1cc57e8a75b"
server_key_iv: "ac63ed697aab5be1c65c8753dc505139"
server_key_version: "encryptionkey@0"
I20260806 10:56:43.402493 12921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:56:43.420933 12921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:56:43.426040 12921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:56:43.426344 12921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:56:43.445726 12921 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.61:33853
I20260806 10:56:43.445770 12987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.61:33853 every 8 connection(s)
I20260806 10:56:43.447398 12921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-1/data/info.pb
I20260806 10:56:43.451746 12988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:56:43.456499 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 12921
I20260806 10:56:43.456725 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-1/wal/instance
I20260806 10:56:43.457247 31592 external_mini_cluster.cc:1428] Setting key 2500c41e2bab0764ebbc9be67dc28d71
I20260806 10:56:43.459251 12988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41563 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:56:43 dist-test-slave-49h3 krb5kdc[12834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013803, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.61@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:56:43.482661 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.462397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52860 (local address 127.30.218.62:41563)
0806 10:56:43.462933 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:43.462938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:43.464223 (+ 1285us) server_negotiation.cc:407] Connection header received
0806 10:56:43.466609 (+ 2386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:43.466615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:43.466715 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:43.466818 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:43.468693 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:43.469586 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:43.471298 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:43.471742 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:43.471866 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:43.476072 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:43.476079 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:43.476082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:43.476128 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:43.479074 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:43.479983 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:43.479987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:43.479990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:43.480144 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:43.480615 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:43.480619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:43.480621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:43.480960 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:43.481124 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:43.481995 (+ 871us) server_negotiation.cc:299] Negotiation successful
0806 10:56:43.482274 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":287,"threads_started":1}
I20260806 10:56:43.483803 12991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.463125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:41563 (local address 127.0.0.1:52860)
0806 10:56:43.464021 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.464076 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:43.466250 (+ 2174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:43.467122 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:43.467132 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:43.467755 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:56:43.468462 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:43.468486 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:43.469772 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:43.469777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:43.471119 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:43.471140 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:43.472014 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:43.472022 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:43.472271 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:43.473000 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:56:43.473024 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:56:43.475766 (+ 2742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:56:43.479263 (+ 3497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:43.479273 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:43.479291 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:43.479837 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:43.480268 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:43.480274 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:43.480276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:43.480485 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:43.481103 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:56:43.481114 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:56:43.481593 (+ 479us) client_negotiation.cc:769] Sending connection context
0806 10:56:43.481881 (+ 288us) client_negotiation.cc:240] Negotiation successful
0806 10:56:43.482503 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":370,"threads_started":1}
WARNING: no policy specified for kudu/127.30.218.60@KRBTEST.COM; defaulting to no policy
I20260806 10:56:43.485707 12988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.60@KRBTEST.COM" created.
Aug 06 10:56:43 dist-test-slave-49h3 krb5kdc[12834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013803, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.61@KRBTEST.COM for kudu/127.30.218.61@KRBTEST.COM
I20260806 10:56:43.500880 12991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.486578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33853 (local address 127.0.0.1:36238)
0806 10:56:43.486838 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.486858 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:43.487008 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:43.487945 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:43.487948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:43.488399 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:56:43.488586 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:43.488613 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:43.490040 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:43.490046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:43.490912 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:43.490924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:43.491788 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:43.491794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:43.491986 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:43.492577 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:56:43.492596 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:56:43.494999 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:56:43.498350 (+ 3351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:43.498356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:43.498360 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:43.498730 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:43.499099 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:43.499104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:43.499108 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:43.499262 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:43.499882 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:56:43.499887 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:56:43.500028 (+ 141us) client_negotiation.cc:769] Sending connection context
0806 10:56:43.500156 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 10:56:43.500475 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 10:56:43.500972 12993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.486810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36238 (local address 127.30.218.61:33853)
0806 10:56:43.487480 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:43.487486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:43.487513 (+ 27us) server_negotiation.cc:407] Connection header received
0806 10:56:43.487572 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:43.487578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:43.487786 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:43.487901 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:43.488767 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:43.489904 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:43.491050 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:43.491646 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:43.491760 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:43.495244 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:43.495261 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:43.495279 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:43.495312 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:43.498203 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:43.498858 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:43.498866 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:43.498868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:43.498991 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:43.499361 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:43.499365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:43.499367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:43.499761 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:43.499971 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:43.500325 (+ 354us) server_negotiation.cc:299] Negotiation successful
0806 10:56:43.500612 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":510,"thread_start_us":384,"threads_started":1}
I20260806 10:56:43.504129 12988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:56:43.505417 12991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.504898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:33733 (local address 127.0.0.1:40180)
0806 10:56:43.505112 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.505250 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 10:56:43.505776 12940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
I20260806 10:56:43.507534 12991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.507152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:33733 (local address 127.0.0.1:40186)
0806 10:56:43.507310 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.507405 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 10:56:43.507763 12988 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:33733: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.60.keytab.
Entry for principal kudu/127.30.218.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.60.keytab.
I20260806 10:56:43.523253 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.60:33733
--webserver_interface=127.30.218.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.60.keytab
--principal=kudu/127.30.218.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:38323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:41563,127.30.218.61:33853,127.30.218.60:33733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:56:43.557086 12988 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } attempt: 1
I20260806 10:56:43.558256 12991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.557787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:33733 (local address 127.0.0.1:40188)
0806 10:56:43.557991 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.558100 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 10:56:43.559333 12991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.558997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:33733 (local address 127.0.0.1:40194)
0806 10:56:43.559146 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.559221 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 10:56:43.559541 12988 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:33733: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
I20260806 10:56:43.577255 12914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } attempt: 5
Aug 06 10:56:43 dist-test-slave-49h3 krb5kdc[12834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013802, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for kudu/127.30.218.61@KRBTEST.COM
I20260806 10:56:43.590133 12993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.578118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36246 (local address 127.30.218.61:33853)
0806 10:56:43.578299 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:43.578305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:43.578384 (+ 79us) server_negotiation.cc:407] Connection header received
0806 10:56:43.578604 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:43.578608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:43.578674 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:43.578760 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:43.579595 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:43.580283 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:43.581515 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:43.581951 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:43.582071 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:43.585084 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:43.585088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:43.585091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:43.585123 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:43.587615 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:43.588228 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:43.588232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:43.588234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:43.588354 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:43.588715 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:43.588719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:43.588723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:43.589024 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:43.589158 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:43.589601 (+ 443us) server_negotiation.cc:299] Negotiation successful
0806 10:56:43.589811 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 10:56:43.590129 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.578061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33853 (local address 127.0.0.1:36246)
0806 10:56:43.578287 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.578309 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:43.578453 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:43.578817 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:43.578822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:43.579240 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:56:43.579426 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:43.579436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:43.580421 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:43.580426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:43.581349 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:43.581362 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:43.582103 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:43.582109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:43.582280 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:43.582891 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:56:43.582907 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:56:43.584895 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:56:43.587745 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:43.587751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:43.587755 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:43.588117 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:43.588462 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:43.588467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:43.588471 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:43.588614 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:43.589140 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:56:43.589145 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:56:43.589298 (+ 153us) client_negotiation.cc:769] Sending connection context
0806 10:56:43.589475 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 10:56:43.589770 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 10:56:43.591369 12914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:56:43.592561 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.592169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:33733 (local address 127.0.0.1:40196)
0806 10:56:43.592329 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.592406 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 10:56:43.593541 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.593173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:33733 (local address 127.0.0.1:40200)
0806 10:56:43.593328 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.593394 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 10:56:43.593748 12914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:33733: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
I20260806 10:56:43.627770 12988 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } attempt: 2
I20260806 10:56:43.629045 12991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.628576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:33733 (local address 127.0.0.1:40214)
0806 10:56:43.628779 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.628874 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 10:56:43.630002 12991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.629681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:33733 (local address 127.0.0.1:40226)
0806 10:56:43.629819 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.629886 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 10:56:43.630203 12988 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:33733: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
I20260806 10:56:43.644979 12914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } attempt: 1
I20260806 10:56:43.646284 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.645801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:33733 (local address 127.0.0.1:40234)
0806 10:56:43.646000 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.646123 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 10:56:43.647248 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.646933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:33733 (local address 127.0.0.1:40250)
0806 10:56:43.647076 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.647146 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 10:56:43.647445 12914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:33733: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
I20260806 10:56:43.715689 12914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } attempt: 2
I20260806 10:56:43.716950 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.716397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:33733 (local address 127.0.0.1:40262)
0806 10:56:43.716643 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.716779 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 10:56:43.718197 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.717794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:33733 (local address 127.0.0.1:40266)
0806 10:56:43.717958 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.718055 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 10:56:43.718428 12914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:33733: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
I20260806 10:56:43.721439 12988 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } attempt: 3
I20260806 10:56:43.722651 12991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.722147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:33733 (local address 127.0.0.1:40276)
0806 10:56:43.722368 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.722465 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 10:56:43.723744 12991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.723272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:33733 (local address 127.0.0.1:40286)
0806 10:56:43.723513 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.723611 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260806 10:56:43.723982 12988 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:33733: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
I20260806 10:56:43.824641 12914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } attempt: 3
I20260806 10:56:43.825920 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.825328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:33733 (local address 127.0.0.1:40292)
0806 10:56:43.825599 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.825723 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260806 10:56:43.827167 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.826721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:33733 (local address 127.0.0.1:40300)
0806 10:56:43.826932 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.827032 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260806 10:56:43.827417 12914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:33733: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
I20260806 10:56:43.867188 12988 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } attempt: 4
I20260806 10:56:43.868281 12991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.867805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:33733 (local address 127.0.0.1:40304)
0806 10:56:43.868031 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.868117 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 10:56:43.869315 12991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:43.869007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:33733 (local address 127.0.0.1:40310)
0806 10:56:43.869133 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:43.869207 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 10:56:43.869558 12988 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:33733: Network error: Client connection negotiation failed: client connection to 127.30.218.60:33733: connect: Connection refused (error 111)
W20260806 10:56:43.891472 12996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:56:43.891803 12996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:56:43.891934 12996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:56:43.899488 12996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:56:43.899626 12996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:56:43.899684 12996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:56:43.899775 12996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:56:43.899834 12996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:56:43.910638 12996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:38323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:41563,127.30.218.61:33853,127.30.218.60:33733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.60.keytab
--principal=kudu/127.30.218.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.60:33733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.218.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:56:43.912124 12996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:56:43.913508 12996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:56:43.925385 13002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:43.925387 13004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:43.925503 13001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:43.926512 12996 server_base.cc:1034] running on GCE node
I20260806 10:56:43.927382 12996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:56:43.929011 12996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:56:43.930255 12996 hybrid_clock.cc:648] HybridClock initialized: now 1786013803930202 us; error 45 us; skew 500 ppm
I20260806 10:56:43.930634 12996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:56:43 dist-test-slave-49h3 krb5kdc[12834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013803, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:43.933974 12996 init.cc:377] Logged in from keytab as kudu/127.30.218.60@KRBTEST.COM (short username kudu)
I20260806 10:56:43.935850 12996 webserver.cc:460] Webserver started at http://127.30.218.60:36767/ using document root <none> and password file <none>
I20260806 10:56:43.936815 12996 fs_manager.cc:359] Metadata directory not provided
I20260806 10:56:43.936967 12996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:56:43.937357 12996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:56:43.941774 12996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-2/data/instance:
uuid: "f3499693797d448c973bd86210b9c09d"
format_stamp: "Formatted at 2026-08-06 10:56:43 on dist-test-slave-49h3"
server_key: "89a493de8e233be3b0b8a2f5cd91a76c"
server_key_iv: "5ed54c4f881fb6f4b638ad35f5b44f5e"
server_key_version: "encryptionkey@0"
I20260806 10:56:43.942993 12996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-2/wal/instance:
uuid: "f3499693797d448c973bd86210b9c09d"
format_stamp: "Formatted at 2026-08-06 10:56:43 on dist-test-slave-49h3"
server_key: "89a493de8e233be3b0b8a2f5cd91a76c"
server_key_iv: "5ed54c4f881fb6f4b638ad35f5b44f5e"
server_key_version: "encryptionkey@0"
I20260806 10:56:43.948792 12996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 10:56:43.952803 13012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:43.954447 12996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:56:43.954655 12996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-2/wal
uuid: "f3499693797d448c973bd86210b9c09d"
format_stamp: "Formatted at 2026-08-06 10:56:43 on dist-test-slave-49h3"
server_key: "89a493de8e233be3b0b8a2f5cd91a76c"
server_key_iv: "5ed54c4f881fb6f4b638ad35f5b44f5e"
server_key_version: "encryptionkey@0"
I20260806 10:56:43.954879 12996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:56:43.970897 12996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:56:43.975348 12996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:56:43.975625 12996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:56:43.993826 12996 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.60:33733
I20260806 10:56:43.993860 13064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.60:33733 every 8 connection(s)
I20260806 10:56:43.995597 12996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-2/data/info.pb
I20260806 10:56:44.000452 13065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:56:44.002799 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 12996
I20260806 10:56:44.003038 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786013589554914-31592-0/minicluster-data/master-2/wal/instance
I20260806 10:56:44.003628 31592 external_mini_cluster.cc:1428] Setting key a38eb9f4a40911c99a9288dfe7bb8d46
I20260806 10:56:44.004971 12914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } attempt: 4
I20260806 10:56:44.012167 13065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41563 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:56:44 dist-test-slave-49h3 krb5kdc[12834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013802, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for kudu/127.30.218.60@KRBTEST.COM
Aug 06 10:56:44 dist-test-slave-49h3 krb5kdc[12834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.60@KRBTEST.COM
Aug 06 10:56:44 dist-test-slave-49h3 krb5kdc[12834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013803, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.60@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:56:44.043701 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:44.005733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:33733 (local address 127.0.0.1:40332)
0806 10:56:44.005954 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:44.005980 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:44.006153 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:44.025189 (+ 19036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:44.025195 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:44.025808 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:56:44.026063 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:44.026079 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.028770 (+ 2691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.028777 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:44.030010 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:44.030025 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.031423 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.031430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:44.031646 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:44.032400 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:56:44.032426 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:56:44.035046 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:56:44.039684 (+ 4638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:44.039694 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:44.039699 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:44.040356 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:44.041002 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:44.041008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:44.041012 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:44.041198 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:44.042530 (+ 1332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:56:44.042535 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:56:44.042763 (+ 228us) client_negotiation.cc:769] Sending connection context
0806 10:56:44.042997 (+ 234us) client_negotiation.cc:240] Negotiation successful
0806 10:56:44.043317 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":41}
I20260806 10:56:44.045145 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:44.006447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40332 (local address 127.30.218.60:33733)
0806 10:56:44.023020 (+ 16573us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:44.023026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:44.023062 (+ 36us) server_negotiation.cc:407] Connection header received
0806 10:56:44.024185 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:44.024214 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:44.024737 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:44.025125 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:44.026303 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.028521 (+ 2218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.030167 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.031204 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.031397 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:44.035320 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:44.035333 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:44.035352 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:44.035392 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:44.039475 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:44.040548 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:44.040558 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:44.040571 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:44.040837 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:44.041374 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:44.041379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:44.041382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:44.042343 (+ 961us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:44.042560 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:44.043316 (+ 756us) server_negotiation.cc:299] Negotiation successful
0806 10:56:44.043818 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":113,"server-negotiator.queue_time_us":16383,"spinlock_wait_cycles":49664,"thread_start_us":13783,"threads_started":1}
I20260806 10:56:44.045192 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:44.006447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40318 (local address 127.30.218.60:33733)
0806 10:56:44.020367 (+ 13920us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:44.020390 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:44.021397 (+ 1007us) server_negotiation.cc:407] Connection header received
0806 10:56:44.024175 (+ 2778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:44.024210 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:44.024735 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:44.025127 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:44.028748 (+ 3621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.029853 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.031819 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.032385 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.032536 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:44.037721 (+ 5185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:44.037726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:44.037730 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:44.037772 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:44.040822 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:44.041763 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:44.041767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:44.041769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:44.041885 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:44.042286 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:44.042289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:44.042291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:44.042625 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:44.042898 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:44.043941 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0806 10:56:44.044158 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":712,"server-negotiator.queue_time_us":13534,"spinlock_wait_cycles":40192,"thread_start_us":16060,"threads_started":1}
I20260806 10:56:44.046772 13071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:44.017242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52870 (local address 127.30.218.62:41563)
0806 10:56:44.018588 (+ 1346us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:44.018596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:44.027534 (+ 8938us) server_negotiation.cc:407] Connection header received
0806 10:56:44.028770 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:44.028777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:44.028868 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:44.029011 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:44.030720 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.031722 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.033807 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.034492 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.034632 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:44.039203 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:44.039211 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:44.039217 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:44.039268 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:44.042809 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:44.043756 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:44.043761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:44.043765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:44.043922 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:44.044369 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:44.044375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:44.044379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:44.044739 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:44.044886 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:44.046115 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0806 10:56:44.046380 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":1168,"thread_start_us":987,"threads_started":1}
I20260806 10:56:44.047876 13070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:44.016463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:41563 (local address 127.0.0.1:52870)
0806 10:56:44.027372 (+ 10909us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:44.027426 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:44.028469 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:44.029108 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:44.029117 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:44.030130 (+ 1013us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:56:44.030528 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:44.030542 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.031899 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.031905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:44.033577 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:44.033593 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.034789 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.034798 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:44.034999 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:44.035696 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:56:44.035721 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:56:44.038871 (+ 3150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:56:44.042974 (+ 4103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:44.042986 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:44.042994 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:44.043603 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:44.044053 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:44.044058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:44.044061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:44.044230 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:44.044958 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:56:44.044965 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:56:44.045220 (+ 255us) client_negotiation.cc:769] Sending connection context
0806 10:56:44.045952 (+ 732us) client_negotiation.cc:240] Negotiation successful
0806 10:56:44.046325 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10608,"thread_start_us":11074,"threads_started":1}
I20260806 10:56:44.049302 13065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:56:44.054080 12914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69bf9380878147df8ee0b27298cd74c6: Bootstrap starting.
Aug 06 10:56:44 dist-test-slave-49h3 krb5kdc[12834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013803, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.60@KRBTEST.COM for kudu/127.30.218.61@KRBTEST.COM
I20260806 10:56:44.058701 12914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69bf9380878147df8ee0b27298cd74c6: Neither blocks nor log segments found. Creating new log.
Aug 06 10:56:44 dist-test-slave-49h3 krb5kdc[12834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:56:44.060165 12914 log.cc:824] T 00000000000000000000000000000000 P 69bf9380878147df8ee0b27298cd74c6: Log is configured to *not* fsync() on all Append() calls
I20260806 10:56:44.063802 12914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69bf9380878147df8ee0b27298cd74c6: No bootstrap required, opened a new log
I20260806 10:56:44.066349 13070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:44.049958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33853 (local address 127.0.0.1:36258)
0806 10:56:44.050148 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:44.050165 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:44.050255 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:44.051290 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:44.051295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:44.051932 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:56:44.052121 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:44.052144 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.053307 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.053313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:44.054306 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:44.054323 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.055030 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.055037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:44.055234 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:44.055984 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:56:44.056011 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:56:44.059028 (+ 3017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:56:44.063196 (+ 4168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:44.063204 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:44.063208 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:44.063805 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:44.064337 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:44.064342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:44.064346 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:44.064550 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:44.065228 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:56:44.065233 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:56:44.065403 (+ 170us) client_negotiation.cc:769] Sending connection context
0806 10:56:44.065610 (+ 207us) client_negotiation.cc:240] Negotiation successful
0806 10:56:44.065959 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 10:56:44.066368 12993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:44.050384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36258 (local address 127.30.218.61:33853)
0806 10:56:44.050596 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:44.050603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:44.050621 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:56:44.050735 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:44.050741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:44.050860 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:44.051011 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:44.052309 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.053136 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.054456 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.054878 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.054994 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:44.059383 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:44.059391 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:44.059397 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:44.059449 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:44.063002 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:44.063985 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:44.063991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:44.063995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:44.064185 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:44.064686 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:44.064691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:44.064694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:44.065073 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:44.065273 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:44.065728 (+ 455us) server_negotiation.cc:299] Negotiation successful
0806 10:56:44.065961 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 10:56:44.067723 13071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:44.048768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52878 (local address 127.30.218.62:41563)
0806 10:56:44.048956 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:44.048960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:44.049820 (+ 860us) server_negotiation.cc:407] Connection header received
0806 10:56:44.050000 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:44.050005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:44.050091 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:44.050207 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:44.051625 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.052615 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.054563 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.055307 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.055442 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:44.061103 (+ 5661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:44.061110 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:44.061114 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:44.061152 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:44.064307 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:44.065197 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:44.065202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:44.065206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:44.065354 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:44.065851 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:44.065861 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:44.065865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:44.066284 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:44.066585 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:44.067141 (+ 556us) server_negotiation.cc:299] Negotiation successful
0806 10:56:44.067390 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 10:56:44.067911 13065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:56:44.074183 12914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69bf9380878147df8ee0b27298cd74c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf9380878147df8ee0b27298cd74c6" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41563 } } peers { permanent_uuid: "6b2d1625b6a949119fbdf233d7610ae8" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } } peers { permanent_uuid: "f3499693797d448c973bd86210b9c09d" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } }
I20260806 10:56:44.074600 12914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69bf9380878147df8ee0b27298cd74c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:56:44.074730 12914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69bf9380878147df8ee0b27298cd74c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69bf9380878147df8ee0b27298cd74c6, State: Initialized, Role: FOLLOWER
I20260806 10:56:44.075428 12914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69bf9380878147df8ee0b27298cd74c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf9380878147df8ee0b27298cd74c6" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41563 } } peers { permanent_uuid: "6b2d1625b6a949119fbdf233d7610ae8" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } } peers { permanent_uuid: "f3499693797d448c973bd86210b9c09d" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } }
Aug 06 10:56:44 dist-test-slave-49h3 krb5kdc[12834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013803, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.60@KRBTEST.COM for kudu/127.30.218.60@KRBTEST.COM
I20260806 10:56:44.076581 13073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69bf9380878147df8ee0b27298cd74c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf9380878147df8ee0b27298cd74c6" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41563 } } peers { permanent_uuid: "6b2d1625b6a949119fbdf233d7610ae8" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } } peers { permanent_uuid: "f3499693797d448c973bd86210b9c09d" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } } }
I20260806 10:56:44.077159 13073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69bf9380878147df8ee0b27298cd74c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:56:44.078109 12914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69bf9380878147df8ee0b27298cd74c6 [sys.catalog]: configured and running, proceeding with master startup.
Aug 06 10:56:44 dist-test-slave-49h3 krb5kdc[12834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.61@KRBTEST.COM
I20260806 10:56:44.083388 13070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:44.068794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:33733 (local address 127.0.0.1:40334)
0806 10:56:44.068922 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:44.068941 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:44.069046 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:44.069373 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:44.069378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:44.069880 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:56:44.070082 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:44.070095 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.071669 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.071675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:44.072810 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:44.072823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.073554 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.073559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:44.073772 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:44.074415 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:56:44.074437 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:56:44.077168 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:56:44.080512 (+ 3344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:44.080528 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:44.080533 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:44.080928 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:44.081293 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:44.081298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:44.081302 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:44.081480 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:44.082304 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:56:44.082308 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:56:44.082465 (+ 157us) client_negotiation.cc:769] Sending connection context
0806 10:56:44.082635 (+ 170us) client_negotiation.cc:240] Negotiation successful
0806 10:56:44.082989 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260806 10:56:44.083647 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:44.068736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40334 (local address 127.30.218.60:33733)
0806 10:56:44.068987 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:44.068993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:44.069012 (+ 19us) server_negotiation.cc:407] Connection header received
0806 10:56:44.069157 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:44.069162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:44.069234 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:44.069332 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:44.070253 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.071516 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.072937 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.073360 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.073492 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:44.077385 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:44.077391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:44.077394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:44.077426 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:44.080378 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:44.081053 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:44.081059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:44.081063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:44.081189 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:44.081609 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:44.081614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:44.081617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:44.082077 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:44.082216 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:44.082796 (+ 580us) server_negotiation.cc:299] Negotiation successful
0806 10:56:44.083225 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":113,"server-negotiator.queue_time_us":111}
I20260806 10:56:44.090708 12993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:44.070585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36264 (local address 127.30.218.61:33853)
0806 10:56:44.070746 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:44.070750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:44.070764 (+ 14us) server_negotiation.cc:407] Connection header received
0806 10:56:44.070804 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:44.070808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:44.070861 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:44.070942 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:44.072488 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.073591 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.075351 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.075951 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.077575 (+ 1624us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:44.082053 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:44.082061 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:44.082065 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:44.082111 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:44.085907 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:44.087053 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:44.087059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:44.087063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:44.087248 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:44.087731 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:44.087736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:44.087739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:44.088143 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:44.088483 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:44.089949 (+ 1466us) server_negotiation.cc:299] Negotiation successful
0806 10:56:44.090276 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 10:56:44.091391 13065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d: Bootstrap starting.
I20260806 10:56:44.095728 13065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d: Neither blocks nor log segments found. Creating new log.
W20260806 10:56:44.096297 12868 tablet.cc:2369] T 00000000000000000000000000000000 P 69bf9380878147df8ee0b27298cd74c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:56:44.097149 13065 log.cc:824] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d: Log is configured to *not* fsync() on all Append() calls
I20260806 10:56:44.103029 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260806 10:56:44.103281 13084 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 69bf9380878147df8ee0b27298cd74c6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 10:56:44.103431 13084 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 10:56:44.091130 12847 master_runner.cc:186] Error getting master registration for 127.30.218.61:33853: OK, instance_id { permanent_uuid: "6b2d1625b6a949119fbdf233d7610ae8" instance_seqno: 1786013803440257 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 10:56:44.105381 12847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 10:56:44.108963 13065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d: No bootstrap required, opened a new log
I20260806 10:56:44.120255 13065 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf9380878147df8ee0b27298cd74c6" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41563 } } peers { permanent_uuid: "6b2d1625b6a949119fbdf233d7610ae8" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } } peers { permanent_uuid: "f3499693797d448c973bd86210b9c09d" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } }
I20260806 10:56:44.120770 13065 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:56:44.120903 13065 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3499693797d448c973bd86210b9c09d, State: Initialized, Role: FOLLOWER
I20260806 10:56:44.121655 13065 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf9380878147df8ee0b27298cd74c6" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41563 } } peers { permanent_uuid: "6b2d1625b6a949119fbdf233d7610ae8" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } } peers { permanent_uuid: "f3499693797d448c973bd86210b9c09d" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } }
I20260806 10:56:44.128882 13093 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf9380878147df8ee0b27298cd74c6" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41563 } } peers { permanent_uuid: "6b2d1625b6a949119fbdf233d7610ae8" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } } peers { permanent_uuid: "f3499693797d448c973bd86210b9c09d" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } } }
I20260806 10:56:44.129436 13093 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:56:44.130604 13065 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:56:44.135157 13093 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:56:44.135519 13093 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf9380878147df8ee0b27298cd74c6" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41563 } } peers { permanent_uuid: "6b2d1625b6a949119fbdf233d7610ae8" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } } peers { permanent_uuid: "f3499693797d448c973bd86210b9c09d" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } }
I20260806 10:56:44.139472 12889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f3499693797d448c973bd86210b9c09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69bf9380878147df8ee0b27298cd74c6" is_pre_election: true
I20260806 10:56:44.140242 12889 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 69bf9380878147df8ee0b27298cd74c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3499693797d448c973bd86210b9c09d in term 0.
I20260806 10:56:44.141037 13014 leader_election.cc:302] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69bf9380878147df8ee0b27298cd74c6, f3499693797d448c973bd86210b9c09d; no voters:
I20260806 10:56:44.150173 13096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:56:44.150451 13096 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:56:44.150624 13096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:56:44.158440 13093 leader_election.cc:288] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69bf9380878147df8ee0b27298cd74c6 (127.30.218.62:41563), 6b2d1625b6a949119fbdf233d7610ae8 (127.30.218.61:33853)
I20260806 10:56:44.162220 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:44.119753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40350 (local address 127.30.218.60:33733)
0806 10:56:44.120190 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:44.120197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:44.133372 (+ 13175us) server_negotiation.cc:407] Connection header received
0806 10:56:44.133976 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:44.133982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:44.134093 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:44.134168 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:44.137060 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.138146 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.140245 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.140912 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.141008 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:44.150916 (+ 9908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:44.150925 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:44.150929 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:44.150984 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:44.154820 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:44.156946 (+ 2126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:44.156952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:44.156955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:44.157146 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:44.158065 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:44.158070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:44.158074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:44.158433 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:44.158822 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:44.161281 (+ 2459us) server_negotiation.cc:299] Negotiation successful
0806 10:56:44.161567 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294}
I20260806 10:56:44.163618 13096 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf9380878147df8ee0b27298cd74c6" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41563 } } peers { permanent_uuid: "6b2d1625b6a949119fbdf233d7610ae8" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } } peers { permanent_uuid: "f3499693797d448c973bd86210b9c09d" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } }
I20260806 10:56:44.165520 12889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f3499693797d448c973bd86210b9c09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69bf9380878147df8ee0b27298cd74c6"
I20260806 10:56:44.165791 12889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69bf9380878147df8ee0b27298cd74c6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:56:44.168756 12988 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } attempt: 5
I20260806 10:56:44.169699 13096 leader_election.cc:288] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [CANDIDATE]: Term 1 election: Requested vote from peers 69bf9380878147df8ee0b27298cd74c6 (127.30.218.62:41563), 6b2d1625b6a949119fbdf233d7610ae8 (127.30.218.61:33853)
I20260806 10:56:44.171212 12889 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 69bf9380878147df8ee0b27298cd74c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3499693797d448c973bd86210b9c09d in term 1.
I20260806 10:56:44.171916 13014 leader_election.cc:302] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69bf9380878147df8ee0b27298cd74c6, f3499693797d448c973bd86210b9c09d; no voters:
I20260806 10:56:44.172513 13096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:56:44.173017 13096 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [term 1 LEADER]: Becoming Leader. State: Replica: f3499693797d448c973bd86210b9c09d, State: Running, Role: LEADER
I20260806 10:56:44.174423 13096 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf9380878147df8ee0b27298cd74c6" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41563 } } peers { permanent_uuid: "6b2d1625b6a949119fbdf233d7610ae8" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } } peers { permanent_uuid: "f3499693797d448c973bd86210b9c09d" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } }
I20260806 10:56:44.179610 13071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:44.119971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52882 (local address 127.30.218.62:41563)
0806 10:56:44.120360 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:44.120366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:44.130631 (+ 10265us) server_negotiation.cc:407] Connection header received
0806 10:56:44.131814 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:44.131822 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:44.131933 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:44.132023 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:44.153765 (+ 21742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.154952 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.159162 (+ 4210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.159886 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.160018 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:44.167903 (+ 7885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:44.167911 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:44.167916 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:44.167957 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:44.171413 (+ 3456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:44.174217 (+ 2804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:44.174224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:44.174228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:44.174418 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:44.174931 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:44.174936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:44.174940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:44.175348 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:44.175658 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:44.178637 (+ 2979us) server_negotiation.cc:299] Negotiation successful
0806 10:56:44.179108 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253}
Aug 06 10:56:44 dist-test-slave-49h3 krb5kdc[12834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013803, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.61@KRBTEST.COM for kudu/127.30.218.60@KRBTEST.COM
I20260806 10:56:44.187093 12993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:44.126689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36280 (local address 127.30.218.61:33853)
0806 10:56:44.126945 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:44.126953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:44.143090 (+ 16137us) server_negotiation.cc:407] Connection header received
0806 10:56:44.145854 (+ 2764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:44.145862 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:44.145969 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:44.146070 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:44.148650 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.149900 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.158901 (+ 9001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.159537 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.159645 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:44.166546 (+ 6901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:44.166554 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:44.166559 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:44.166609 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:44.173176 (+ 6567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:44.183957 (+ 10781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:44.183965 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:44.183968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:44.184165 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:44.184933 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:44.184945 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:44.184948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:44.185414 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:44.185707 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:44.186417 (+ 710us) server_negotiation.cc:299] Negotiation successful
0806 10:56:44.186708 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260806 10:56:44.190006 12991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:44.169375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:33733 (local address 127.0.0.1:40360)
0806 10:56:44.169668 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:44.169687 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:44.169808 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:44.170387 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:44.170392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:44.170877 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:56:44.171107 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:44.171121 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.172664 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.172670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:44.174134 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:44.174154 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.175386 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.175393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:44.175642 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:44.176315 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:56:44.176341 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:56:44.182488 (+ 6147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:56:44.186927 (+ 4439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:44.186933 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:44.186937 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:44.187508 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:44.187990 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:44.187995 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:44.187998 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:44.188192 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:44.188959 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:56:44.188963 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:56:44.189146 (+ 183us) client_negotiation.cc:769] Sending connection context
0806 10:56:44.189293 (+ 147us) client_negotiation.cc:240] Negotiation successful
0806 10:56:44.189628 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260806 10:56:44.190850 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:44.169474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40360 (local address 127.30.218.60:33733)
0806 10:56:44.169652 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:44.169658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:44.169927 (+ 269us) server_negotiation.cc:407] Connection header received
0806 10:56:44.169992 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:44.169997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:44.170093 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:44.170187 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:44.171290 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.172484 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.174309 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:44.175051 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:44.175172 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:44.182760 (+ 7588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:44.182769 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:44.182773 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:44.182821 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:44.186744 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:44.187670 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:44.187676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:44.187679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:44.187847 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:44.188333 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:44.188337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:44.188340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:44.188750 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:44.188898 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:44.190215 (+ 1317us) server_negotiation.cc:299] Negotiation successful
0806 10:56:44.190456 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 10:56:44.191708 13093 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3499693797d448c973bd86210b9c09d. Latest consensus state: current_term: 1 leader_uuid: "f3499693797d448c973bd86210b9c09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf9380878147df8ee0b27298cd74c6" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41563 } } peers { permanent_uuid: "6b2d1625b6a949119fbdf233d7610ae8" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } } peers { permanent_uuid: "f3499693797d448c973bd86210b9c09d" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } } }
I20260806 10:56:44.192065 13093 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [sys.catalog]: This master's current role is: LEADER
I20260806 10:56:44.200847 12988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2d1625b6a949119fbdf233d7610ae8: Bootstrap starting.
I20260806 10:56:44.202841 13101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:56:44.221258 12988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b2d1625b6a949119fbdf233d7610ae8: Neither blocks nor log segments found. Creating new log.
W20260806 10:56:44.228056 13018 tablet.cc:2369] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:56:44.233637 12988 log.cc:824] T 00000000000000000000000000000000 P 6b2d1625b6a949119fbdf233d7610ae8: Log is configured to *not* fsync() on all Append() calls
I20260806 10:56:44.236320 13101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:56:44.253651 12988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2d1625b6a949119fbdf233d7610ae8: No bootstrap required, opened a new log
I20260806 10:56:44.285265 12988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b2d1625b6a949119fbdf233d7610ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf9380878147df8ee0b27298cd74c6" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41563 } } peers { permanent_uuid: "6b2d1625b6a949119fbdf233d7610ae8" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } } peers { permanent_uuid: "f3499693797d448c973bd86210b9c09d" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } }
I20260806 10:56:44.290146 12988 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b2d1625b6a949119fbdf233d7610ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:56:44.290328 12988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b2d1625b6a949119fbdf233d7610ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2d1625b6a949119fbdf233d7610ae8, State: Initialized, Role: FOLLOWER
I20260806 10:56:44.291180 12988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b2d1625b6a949119fbdf233d7610ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf9380878147df8ee0b27298cd74c6" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41563 } } peers { permanent_uuid: "6b2d1625b6a949119fbdf233d7610ae8" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } } peers { permanent_uuid: "f3499693797d448c973bd86210b9c09d" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } }
I20260806 10:56:44.286964 12996 master_runner.cc:186] Error getting master registration for 127.30.218.61:33853: OK, instance_id { permanent_uuid: "6b2d1625b6a949119fbdf233d7610ae8" instance_seqno: 1786013803440257 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 10:56:44.309818 12996 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 10:56:44.310557 13113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2d1625b6a949119fbdf233d7610ae8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf9380878147df8ee0b27298cd74c6" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41563 } } peers { permanent_uuid: "6b2d1625b6a949119fbdf233d7610ae8" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } } peers { permanent_uuid: "f3499693797d448c973bd86210b9c09d" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } } }
I20260806 10:56:44.311030 13113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2d1625b6a949119fbdf233d7610ae8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:56:44.311215 12889 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 69bf9380878147df8ee0b27298cd74c6 [term 1 FOLLOWER]: Refusing update from remote peer f3499693797d448c973bd86210b9c09d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:56:44.312321 12962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f3499693797d448c973bd86210b9c09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2d1625b6a949119fbdf233d7610ae8"
I20260806 10:56:44.312491 13096 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69bf9380878147df8ee0b27298cd74c6" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:56:44.312881 12962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b2d1625b6a949119fbdf233d7610ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:56:44.312316 12988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b2d1625b6a949119fbdf233d7610ae8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:56:44.312366 12963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f3499693797d448c973bd86210b9c09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2d1625b6a949119fbdf233d7610ae8" is_pre_election: true
I20260806 10:56:44.317039 12962 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b2d1625b6a949119fbdf233d7610ae8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3499693797d448c973bd86210b9c09d in term 1.
I20260806 10:56:44.321080 12961 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6b2d1625b6a949119fbdf233d7610ae8 [term 1 FOLLOWER]: Refusing update from remote peer f3499693797d448c973bd86210b9c09d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:56:44.324203 13096 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b2d1625b6a949119fbdf233d7610ae8" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:56:44.355435 13113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2d1625b6a949119fbdf233d7610ae8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3499693797d448c973bd86210b9c09d. Latest consensus state: current_term: 1 leader_uuid: "f3499693797d448c973bd86210b9c09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf9380878147df8ee0b27298cd74c6" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41563 } } peers { permanent_uuid: "6b2d1625b6a949119fbdf233d7610ae8" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } } peers { permanent_uuid: "f3499693797d448c973bd86210b9c09d" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } } }
I20260806 10:56:44.355849 13113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2d1625b6a949119fbdf233d7610ae8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:56:44.357385 13111 mvcc.cc:205] Tried to move back new op lower bound from 7315512542431899648 to 7315512541910200320. Current Snapshot: MvccSnapshot[applied={T|T < 7315512542431899648}]
I20260806 10:56:44.360242 13113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2d1625b6a949119fbdf233d7610ae8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3499693797d448c973bd86210b9c09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf9380878147df8ee0b27298cd74c6" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41563 } } peers { permanent_uuid: "6b2d1625b6a949119fbdf233d7610ae8" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } } peers { permanent_uuid: "f3499693797d448c973bd86210b9c09d" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } } }
I20260806 10:56:44.360556 13113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2d1625b6a949119fbdf233d7610ae8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:56:44.366159 13116 mvcc.cc:205] Tried to move back new op lower bound from 7315512542431899648 to 7315512541910200320. Current Snapshot: MvccSnapshot[applied={T|T < 7315512542431899648 or (T in {7315512542431899648})}]
I20260806 10:56:44.373816 13093 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3499693797d448c973bd86210b9c09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf9380878147df8ee0b27298cd74c6" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41563 } } peers { permanent_uuid: "6b2d1625b6a949119fbdf233d7610ae8" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } } peers { permanent_uuid: "f3499693797d448c973bd86210b9c09d" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } } }
I20260806 10:56:44.374205 13093 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [sys.catalog]: This master's current role is: LEADER
I20260806 10:56:44.378762 13101 catalog_manager.cc:1347] Generated new cluster ID: 3753d578a0894d92a233a47050f8282b
I20260806 10:56:44.378896 13101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:56:44.380357 13073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69bf9380878147df8ee0b27298cd74c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3499693797d448c973bd86210b9c09d. Latest consensus state: current_term: 1 leader_uuid: "f3499693797d448c973bd86210b9c09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf9380878147df8ee0b27298cd74c6" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41563 } } peers { permanent_uuid: "6b2d1625b6a949119fbdf233d7610ae8" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } } peers { permanent_uuid: "f3499693797d448c973bd86210b9c09d" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } } }
I20260806 10:56:44.380736 13073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69bf9380878147df8ee0b27298cd74c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:56:44.384721 13073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69bf9380878147df8ee0b27298cd74c6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3499693797d448c973bd86210b9c09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf9380878147df8ee0b27298cd74c6" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41563 } } peers { permanent_uuid: "6b2d1625b6a949119fbdf233d7610ae8" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } } peers { permanent_uuid: "f3499693797d448c973bd86210b9c09d" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } } }
I20260806 10:56:44.385001 13073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69bf9380878147df8ee0b27298cd74c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:56:44.384778 13096 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3499693797d448c973bd86210b9c09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69bf9380878147df8ee0b27298cd74c6" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41563 } } peers { permanent_uuid: "6b2d1625b6a949119fbdf233d7610ae8" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33853 } } peers { permanent_uuid: "f3499693797d448c973bd86210b9c09d" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 33733 } } }
I20260806 10:56:44.385185 13096 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d [sys.catalog]: This master's current role is: LEADER
I20260806 10:56:44.425897 13101 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:56:44.427929 13101 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:56:44.438194 13132 catalog_manager.cc:1259] Loaded cluster ID: 3753d578a0894d92a233a47050f8282b
I20260806 10:56:44.438333 13132 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6b2d1625b6a949119fbdf233d7610ae8: loading cluster ID for follower catalog manager: success
I20260806 10:56:44.442389 13132 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6b2d1625b6a949119fbdf233d7610ae8: acquiring CA information for follower catalog manager: success
W20260806 10:56:44.443966 13132 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260806 10:56:44.455960 13101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d: Generated new TSK 0
I20260806 10:56:44.461838 13101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:56:45.105633 13084 catalog_manager.cc:1259] Loaded cluster ID: 3753d578a0894d92a233a47050f8282b
I20260806 10:56:45.105764 13084 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 69bf9380878147df8ee0b27298cd74c6: loading cluster ID for follower catalog manager: success
I20260806 10:56:45.108507 13084 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 69bf9380878147df8ee0b27298cd74c6: acquiring CA information for follower catalog manager: success
I20260806 10:56:45.110534 13084 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 69bf9380878147df8ee0b27298cd74c6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 10:56:45.447675 13132 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6b2d1625b6a949119fbdf233d7610ae8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 10:56:45.616261 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 12847
I20260806 10:56:45.618906 12909 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:56:45.745566 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 12847
I20260806 10:56:45.772158 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 12921
I20260806 10:56:45.773572 12983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:56:45.873759 13139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:45.872709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:41563 (local address 127.0.0.1:57986)
0806 10:56:45.873278 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:45.873512 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:41563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":310,"threads_started":1}
W20260806 10:56:45.874137 13014 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.218.62:41563: connect: Connection refused (error 111)
I20260806 10:56:45.883011 13139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:45.882531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:41563 (local address 127.0.0.1:57996)
0806 10:56:45.882717 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:45.882827 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:41563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 10:56:45.883256 13014 consensus_peers.cc:487] T 00000000000000000000000000000000 P f3499693797d448c973bd86210b9c09d -> Peer 69bf9380878147df8ee0b27298cd74c6 (127.30.218.62:41563): Couldn't send request to peer 69bf9380878147df8ee0b27298cd74c6. Status: Network error: Client connection negotiation failed: client connection to 127.30.218.62:41563: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 10:56:45.897876 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 12921
I20260806 10:56:45.923512 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 12996
I20260806 10:56:45.924996 13060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:56:46.051409 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 12996
2026-08-06T10:56:46Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5123 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:56:46 dist-test-slave-49h3 krb5kdc[13144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:56:46 dist-test-slave-49h3 krb5kdc[13144](info): set up 2 sockets
Aug 06 10:56:46 dist-test-slave-49h3 krb5kdc[13144](info): commencing operation
krb5kdc: starting...
W20260806 10:56:47.147173 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:56:47 dist-test-slave-49h3 krb5kdc[13144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:56:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:56:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:56:47.493963 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:38921
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:33207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:38921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:56:47.881009 13157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:56:47.881374 13157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:56:47.881541 13157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:56:47.889277 13157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:56:47.889395 13157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:56:47.889482 13157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:56:47.889573 13157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:56:47.889638 13157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:56:47.900395 13157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:33207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:38921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:38921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:56:47.902182 13157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:56:47.903887 13157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:56:47.918287 13162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:47.918221 13165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:47.918270 13163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:47.919658 13157 server_base.cc:1034] running on GCE node
I20260806 10:56:47.920635 13157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:56:47.922569 13157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:56:47.923789 13157 hybrid_clock.cc:648] HybridClock initialized: now 1786013807923721 us; error 57 us; skew 500 ppm
I20260806 10:56:47.924202 13157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:56:47 dist-test-slave-49h3 krb5kdc[13144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013807, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:47.933519 13157 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:56:47.935781 13157 webserver.cc:460] Webserver started at http://127.30.218.62:33663/ using document root <none> and password file <none>
I20260806 10:56:47.936699 13157 fs_manager.cc:359] Metadata directory not provided
I20260806 10:56:47.936861 13157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:56:47.937371 13157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:56:47.942531 13157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "c991b7f5ef72497e8e5559974595f71f"
format_stamp: "Formatted at 2026-08-06 10:56:47 on dist-test-slave-49h3"
server_key: "39dbb6a47e4af7b421de597c09cef402"
server_key_iv: "eed694e672e88567e8159311c2b4169e"
server_key_version: "encryptionkey@0"
I20260806 10:56:47.943661 13157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "c991b7f5ef72497e8e5559974595f71f"
format_stamp: "Formatted at 2026-08-06 10:56:47 on dist-test-slave-49h3"
server_key: "39dbb6a47e4af7b421de597c09cef402"
server_key_iv: "eed694e672e88567e8159311c2b4169e"
server_key_version: "encryptionkey@0"
I20260806 10:56:47.950039 13157 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 10:56:47.954422 13172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:47.956151 13157 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 10:56:47.956348 13157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "c991b7f5ef72497e8e5559974595f71f"
format_stamp: "Formatted at 2026-08-06 10:56:47 on dist-test-slave-49h3"
server_key: "39dbb6a47e4af7b421de597c09cef402"
server_key_iv: "eed694e672e88567e8159311c2b4169e"
server_key_version: "encryptionkey@0"
I20260806 10:56:47.956585 13157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:56:47.971690 13157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:56:47.976565 13157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:56:47.976903 13157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:56:47.997428 13157 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:38921
I20260806 10:56:47.997471 13225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:38921 every 8 connection(s)
I20260806 10:56:47.999241 13157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:56:48.004110 13226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:56:48.009024 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 13157
I20260806 10:56:48.009334 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:56:48.010310 31592 external_mini_cluster.cc:1428] Setting key 13f19c8e5460dd9e0bf4735623e4de28
I20260806 10:56:48.018373 13226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c991b7f5ef72497e8e5559974595f71f: Bootstrap starting.
I20260806 10:56:48.022838 13226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c991b7f5ef72497e8e5559974595f71f: Neither blocks nor log segments found. Creating new log.
I20260806 10:56:48.024188 13226 log.cc:824] T 00000000000000000000000000000000 P c991b7f5ef72497e8e5559974595f71f: Log is configured to *not* fsync() on all Append() calls
I20260806 10:56:48.028254 13226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c991b7f5ef72497e8e5559974595f71f: No bootstrap required, opened a new log
Aug 06 10:56:48 dist-test-slave-49h3 krb5kdc[13144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:56:48.039839 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:48.012815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32820 (local address 127.30.218.62:38921)
0806 10:56:48.013795 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:48.013809 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:48.013862 (+ 53us) server_negotiation.cc:407] Connection header received
0806 10:56:48.016581 (+ 2719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:48.016614 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:48.017118 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:48.017573 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:48.019767 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:48.021237 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:48.023360 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:48.024110 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:48.024269 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:48.029814 (+ 5545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:48.029826 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:48.029848 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:48.029908 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:48.034229 (+ 4321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:48.035071 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:48.035081 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:48.035093 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:48.035352 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:48.035764 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:48.035769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:48.035772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:48.036668 (+ 896us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:48.037066 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:48.037415 (+ 349us) server_negotiation.cc:299] Negotiation successful
0806 10:56:48.038092 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":473,"threads_started":1}
I20260806 10:56:48.039837 13226 raft_consensus.cc:348] T 00000000000000000000000000000000 P c991b7f5ef72497e8e5559974595f71f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c991b7f5ef72497e8e5559974595f71f" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 38921 } }
I20260806 10:56:48.040426 13226 raft_consensus.cc:374] T 00000000000000000000000000000000 P c991b7f5ef72497e8e5559974595f71f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:56:48.040526 13226 raft_consensus.cc:729] T 00000000000000000000000000000000 P c991b7f5ef72497e8e5559974595f71f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c991b7f5ef72497e8e5559974595f71f, State: Initialized, Role: FOLLOWER
I20260806 10:56:48.041229 13226 consensus_queue.cc:260] T 00000000000000000000000000000000 P c991b7f5ef72497e8e5559974595f71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c991b7f5ef72497e8e5559974595f71f" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 38921 } }
I20260806 10:56:48.041549 13226 raft_consensus.cc:388] T 00000000000000000000000000000000 P c991b7f5ef72497e8e5559974595f71f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:56:48.041729 13226 raft_consensus.cc:482] T 00000000000000000000000000000000 P c991b7f5ef72497e8e5559974595f71f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:56:48.041936 13226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c991b7f5ef72497e8e5559974595f71f [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:56:48.045262 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:56:48.045534 13226 raft_consensus.cc:504] T 00000000000000000000000000000000 P c991b7f5ef72497e8e5559974595f71f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c991b7f5ef72497e8e5559974595f71f" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 38921 } }
I20260806 10:56:48.046268 13226 leader_election.cc:302] T 00000000000000000000000000000000 P c991b7f5ef72497e8e5559974595f71f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c991b7f5ef72497e8e5559974595f71f; no voters:
I20260806 10:56:48.047024 13226 leader_election.cc:288] T 00000000000000000000000000000000 P c991b7f5ef72497e8e5559974595f71f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:56:48.047163 13231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c991b7f5ef72497e8e5559974595f71f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:56:48.048385 13231 raft_consensus.cc:686] T 00000000000000000000000000000000 P c991b7f5ef72497e8e5559974595f71f [term 1 LEADER]: Becoming Leader. State: Replica: c991b7f5ef72497e8e5559974595f71f, State: Running, Role: LEADER
I20260806 10:56:48.049083 13231 consensus_queue.cc:237] T 00000000000000000000000000000000 P c991b7f5ef72497e8e5559974595f71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c991b7f5ef72497e8e5559974595f71f" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 38921 } }
I20260806 10:56:48.049681 13226 sys_catalog.cc:564] T 00000000000000000000000000000000 P c991b7f5ef72497e8e5559974595f71f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:56:48.052557 13232 sys_catalog.cc:455] T 00000000000000000000000000000000 P c991b7f5ef72497e8e5559974595f71f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c991b7f5ef72497e8e5559974595f71f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c991b7f5ef72497e8e5559974595f71f" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 38921 } } }
I20260806 10:56:48.053573 13233 sys_catalog.cc:455] T 00000000000000000000000000000000 P c991b7f5ef72497e8e5559974595f71f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c991b7f5ef72497e8e5559974595f71f. Latest consensus state: current_term: 1 leader_uuid: "c991b7f5ef72497e8e5559974595f71f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c991b7f5ef72497e8e5559974595f71f" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 38921 } } }
I20260806 10:56:48.054139 13232 sys_catalog.cc:458] T 00000000000000000000000000000000 P c991b7f5ef72497e8e5559974595f71f [sys.catalog]: This master's current role is: LEADER
I20260806 10:56:48.054162 13233 sys_catalog.cc:458] T 00000000000000000000000000000000 P c991b7f5ef72497e8e5559974595f71f [sys.catalog]: This master's current role is: LEADER
I20260806 10:56:48.058938 13240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:56:48.064378 13240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:56:48.074055 13240 catalog_manager.cc:1347] Generated new cluster ID: a57d72c8cca647739a627c5174c564fe
I20260806 10:56:48.074187 13240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:56:48.083813 13240 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:56:48.085881 13240 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:56:48.097856 13240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c991b7f5ef72497e8e5559974595f71f: Generated new TSK 0
I20260806 10:56:48.099164 13240 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:56:48.107802 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 13157
I20260806 10:56:48.109232 13221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:56:48.209998 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 13157
2026-08-06T10:56:48Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2154 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:56:48 dist-test-slave-49h3 krb5kdc[13254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:56:48 dist-test-slave-49h3 krb5kdc[13254](info): set up 2 sockets
Aug 06 10:56:48 dist-test-slave-49h3 krb5kdc[13254](info): commencing operation
krb5kdc: starting...
W20260806 10:56:49.292356 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.005s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:56:49 dist-test-slave-49h3 krb5kdc[13254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:56:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:56:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:56:49.607170 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:42721
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:43985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:42721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:56:49.985340 13267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:56:49.985741 13267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:56:49.985903 13267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:56:49.993810 13267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:56:49.993954 13267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:56:49.994033 13267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:56:49.994113 13267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:56:49.994174 13267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:56:50.004671 13267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:43985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:42721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:42721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:56:50.006140 13267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:56:50.007493 13267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:56:50.019068 13273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:50.019142 13275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:50.019171 13272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:50.020076 13267 server_base.cc:1034] running on GCE node
I20260806 10:56:50.020860 13267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:56:50.022441 13267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:56:50.023685 13267 hybrid_clock.cc:648] HybridClock initialized: now 1786013810023631 us; error 43 us; skew 500 ppm
I20260806 10:56:50.024067 13267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:56:50 dist-test-slave-49h3 krb5kdc[13254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013810, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:50.027473 13267 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:56:50.029075 13267 webserver.cc:460] Webserver started at http://127.30.218.62:41045/ using document root <none> and password file <none>
I20260806 10:56:50.029867 13267 fs_manager.cc:359] Metadata directory not provided
I20260806 10:56:50.029989 13267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:56:50.030314 13267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:56:50.033635 13267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "1a866dfcd2fd4db3a6082469f1e47e67"
format_stamp: "Formatted at 2026-08-06 10:56:50 on dist-test-slave-49h3"
server_key: "57b918a3d7eeb3c385415ab2b4b7269e"
server_key_iv: "6d4b4e9646b87a0dd2e935c73a20c37e"
server_key_version: "encryptionkey@0"
I20260806 10:56:50.034458 13267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "1a866dfcd2fd4db3a6082469f1e47e67"
format_stamp: "Formatted at 2026-08-06 10:56:50 on dist-test-slave-49h3"
server_key: "57b918a3d7eeb3c385415ab2b4b7269e"
server_key_iv: "6d4b4e9646b87a0dd2e935c73a20c37e"
server_key_version: "encryptionkey@0"
I20260806 10:56:50.040181 13267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 10:56:50.044028 13282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:50.045571 13267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 10:56:50.045746 13267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "1a866dfcd2fd4db3a6082469f1e47e67"
format_stamp: "Formatted at 2026-08-06 10:56:50 on dist-test-slave-49h3"
server_key: "57b918a3d7eeb3c385415ab2b4b7269e"
server_key_iv: "6d4b4e9646b87a0dd2e935c73a20c37e"
server_key_version: "encryptionkey@0"
I20260806 10:56:50.045948 13267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:56:50.061844 13267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:56:50.066392 13267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:56:50.066687 13267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:56:50.084642 13267 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:42721
I20260806 10:56:50.084666 13334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:42721 every 8 connection(s)
I20260806 10:56:50.086042 13267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:56:50.088887 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 13267
I20260806 10:56:50.089124 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:56:50.090065 31592 external_mini_cluster.cc:1428] Setting key 7d933289fdc499e9af6b70989e9d0cb4
I20260806 10:56:50.090211 13335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:56:50.103572 13335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67: Bootstrap starting.
Aug 06 10:56:50 dist-test-slave-49h3 krb5kdc[13254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:56:50.107332 13335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67: Neither blocks nor log segments found. Creating new log.
I20260806 10:56:50.108587 13335 log.cc:824] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67: Log is configured to *not* fsync() on all Append() calls
I20260806 10:56:50.112748 13335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67: No bootstrap required, opened a new log
I20260806 10:56:50.114035 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:50.092228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41000 (local address 127.30.218.62:42721)
0806 10:56:50.092981 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:50.092993 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:50.093038 (+ 45us) server_negotiation.cc:407] Connection header received
0806 10:56:50.095084 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:50.095110 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:50.095539 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:50.095962 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:50.097568 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:50.098932 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:50.100399 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:50.101218 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:50.101419 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:50.106089 (+ 4670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:50.106098 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:50.106121 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:50.106174 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:50.109695 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:50.110528 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:50.110536 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:50.110544 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:50.110729 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:50.111206 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:50.111209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:50.111211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:50.111756 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:50.112008 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:50.112503 (+ 495us) server_negotiation.cc:299] Negotiation successful
0806 10:56:50.112870 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":334,"threads_started":1}
I20260806 10:56:50.120862 13335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a866dfcd2fd4db3a6082469f1e47e67" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42721 } }
I20260806 10:56:50.121306 13335 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:56:50.121418 13335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a866dfcd2fd4db3a6082469f1e47e67, State: Initialized, Role: FOLLOWER
I20260806 10:56:50.122138 13335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a866dfcd2fd4db3a6082469f1e47e67" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42721 } }
I20260806 10:56:50.122380 13335 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:56:50.122520 13335 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:56:50.122682 13335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:56:50.124997 13335 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a866dfcd2fd4db3a6082469f1e47e67" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42721 } }
I20260806 10:56:50.125605 13335 leader_election.cc:302] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a866dfcd2fd4db3a6082469f1e47e67; no voters:
I20260806 10:56:50.126328 13335 leader_election.cc:288] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:56:50.126428 13340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:56:50.127246 13340 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 1 LEADER]: Becoming Leader. State: Replica: 1a866dfcd2fd4db3a6082469f1e47e67, State: Running, Role: LEADER
I20260806 10:56:50.127836 13340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a866dfcd2fd4db3a6082469f1e47e67" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42721 } }
I20260806 10:56:50.128587 13335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:56:50.130847 13341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a866dfcd2fd4db3a6082469f1e47e67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a866dfcd2fd4db3a6082469f1e47e67" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42721 } } }
I20260806 10:56:50.131155 13341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [sys.catalog]: This master's current role is: LEADER
I20260806 10:56:50.130745 13342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a866dfcd2fd4db3a6082469f1e47e67. Latest consensus state: current_term: 1 leader_uuid: "1a866dfcd2fd4db3a6082469f1e47e67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a866dfcd2fd4db3a6082469f1e47e67" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42721 } } }
I20260806 10:56:50.131296 13342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [sys.catalog]: This master's current role is: LEADER
I20260806 10:56:50.132113 13346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:56:50.139290 13346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:56:50.149309 13346 catalog_manager.cc:1347] Generated new cluster ID: 403cc141659e4e7bb759a4497f8007db
I20260806 10:56:50.149477 13346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:56:50.162024 13346 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:56:50.163614 13346 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:56:50.173151 13346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67: Generated new TSK 0
I20260806 10:56:50.174309 13346 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
I20260806 10:56:50.257766 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:42721
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:43985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:56:50.619810 13361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:56:50.620131 13361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:56:50.620258 13361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:56:50.627974 13361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:56:50.628116 13361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:56:50.628280 13361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:56:50.639565 13361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:43985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:42721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:56:50.641093 13361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:56:50.642858 13361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:56:50.657212 13366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:50.657125 13367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:50.657166 13369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:50.657943 13361 server_base.cc:1034] running on GCE node
I20260806 10:56:50.658650 13361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:56:50.659729 13361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:56:50.660949 13361 hybrid_clock.cc:648] HybridClock initialized: now 1786013810660893 us; error 53 us; skew 500 ppm
I20260806 10:56:50.661311 13361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:56:50 dist-test-slave-49h3 krb5kdc[13254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013810, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:50.665179 13361 init.cc:377] Logged in from keytab as kudu/127.30.218.1@KRBTEST.COM (short username kudu)
I20260806 10:56:50.667055 13361 webserver.cc:460] Webserver started at http://127.30.218.1:39225/ using document root <none> and password file <none>
I20260806 10:56:50.667826 13361 fs_manager.cc:359] Metadata directory not provided
I20260806 10:56:50.667965 13361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:56:50.668291 13361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:56:50.671594 13361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "3d2c5277ea7546a3bbdef1d58c93fba2"
format_stamp: "Formatted at 2026-08-06 10:56:50 on dist-test-slave-49h3"
server_key: "7d11dbb0738e614a2868f360eafdc805"
server_key_iv: "f6d5a7dd3dc78cf870bf802703579218"
server_key_version: "encryptionkey@0"
I20260806 10:56:50.672559 13361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "3d2c5277ea7546a3bbdef1d58c93fba2"
format_stamp: "Formatted at 2026-08-06 10:56:50 on dist-test-slave-49h3"
server_key: "7d11dbb0738e614a2868f360eafdc805"
server_key_iv: "f6d5a7dd3dc78cf870bf802703579218"
server_key_version: "encryptionkey@0"
I20260806 10:56:50.678624 13361 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260806 10:56:50.682585 13376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:50.684147 13361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:56:50.684340 13361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "3d2c5277ea7546a3bbdef1d58c93fba2"
format_stamp: "Formatted at 2026-08-06 10:56:50 on dist-test-slave-49h3"
server_key: "7d11dbb0738e614a2868f360eafdc805"
server_key_iv: "f6d5a7dd3dc78cf870bf802703579218"
server_key_version: "encryptionkey@0"
I20260806 10:56:50.684557 13361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:56:50.756723 13361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:56:50.761242 13361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:56:50.761590 13361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:56:50.762634 13361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:56:50.764178 13361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:56:50.764263 13361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:50.764380 13361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:56:50.764433 13361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:50.797113 13361 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:43791
I20260806 10:56:50.797158 13489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:43791 every 8 connection(s)
I20260806 10:56:50.798529 13361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:56:50.801728 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 13361
I20260806 10:56:50.801952 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:56:50.802855 31592 external_mini_cluster.cc:1428] Setting key 573bf19a59a44b600242d94ac0d7e22f
Aug 06 10:56:50 dist-test-slave-49h3 krb5kdc[13254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013810, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:56:50.822816 13492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:50.801303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:44821 (local address 127.30.218.62:42721)
0806 10:56:50.801851 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:50.801858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:50.803224 (+ 1366us) server_negotiation.cc:407] Connection header received
0806 10:56:50.805353 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:50.805357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:50.805429 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:50.805556 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:50.807767 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:50.808538 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:50.810866 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:50.811307 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:50.811418 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:50.816152 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:50.816159 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:50.816162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:50.816208 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:50.818915 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:50.820000 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:50.820008 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:50.820011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:50.820192 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:50.820692 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:50.820697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:50.820700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:50.821116 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:50.821271 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:50.822157 (+ 886us) server_negotiation.cc:299] Negotiation successful
0806 10:56:50.822421 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":263,"threads_started":1}
I20260806 10:56:50.823664 13493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:50.801877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.1:44821)
0806 10:56:50.802798 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:50.802856 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:50.805030 (+ 2174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:50.805845 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:50.805858 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:50.806684 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:56:50.807549 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:50.807577 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:50.808726 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:50.808731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:50.810692 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:50.810712 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:50.811584 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:50.811591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:50.811905 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:50.812739 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:56:50.812772 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:56:50.815808 (+ 3036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:56:50.819111 (+ 3303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:50.819122 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:50.819145 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:50.819855 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:50.820298 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:50.820301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:50.820304 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:50.820551 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:50.821258 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:56:50.821266 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:56:50.821702 (+ 436us) client_negotiation.cc:769] Sending connection context
0806 10:56:50.821983 (+ 281us) client_negotiation.cc:240] Negotiation successful
0806 10:56:50.822466 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":345,"threads_started":1}
I20260806 10:56:50.825408 13490 heartbeater.cc:347] Connected to a master server at 127.30.218.62:42721
I20260806 10:56:50.825801 13490 heartbeater.cc:464] Registering TS with master...
I20260806 10:56:50.826638 13490 heartbeater.cc:511] Master 127.30.218.62:42721 requested a full tablet report, sending...
I20260806 10:56:50.829035 13299 ts_manager.cc:194] Registered new tserver with Master: 3d2c5277ea7546a3bbdef1d58c93fba2 (127.30.218.1:43791)
I20260806 10:56:50.830744 13299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.1@KRBTEST.COM'} at 127.30.218.1:44821
WARNING: no policy specified for kudu/127.30.218.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
I20260806 10:56:50.873468 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:42721
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:43985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:56:51.235932 13496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:56:51.236261 13496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:56:51.236392 13496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:56:51.244017 13496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:56:51.244149 13496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:56:51.244306 13496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:56:51.255208 13496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:43985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:42721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:56:51.256582 13496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:56:51.257954 13496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:56:51.271399 13501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:51.271433 13504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:51.271371 13502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:51.272423 13496 server_base.cc:1034] running on GCE node
I20260806 10:56:51.273128 13496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:56:51.274206 13496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:56:51.275418 13496 hybrid_clock.cc:648] HybridClock initialized: now 1786013811275364 us; error 53 us; skew 500 ppm
I20260806 10:56:51.275756 13496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:56:51 dist-test-slave-49h3 krb5kdc[13254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013811, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:51.279136 13496 init.cc:377] Logged in from keytab as kudu/127.30.218.2@KRBTEST.COM (short username kudu)
I20260806 10:56:51.280718 13496 webserver.cc:460] Webserver started at http://127.30.218.2:35963/ using document root <none> and password file <none>
I20260806 10:56:51.281508 13496 fs_manager.cc:359] Metadata directory not provided
I20260806 10:56:51.281656 13496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:56:51.281970 13496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:56:51.285188 13496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "a2291f6829ab46b084f98b6f7e974db4"
format_stamp: "Formatted at 2026-08-06 10:56:51 on dist-test-slave-49h3"
server_key: "3b1505198463f668cac7d66d6d2bebaf"
server_key_iv: "456df9b2734955974a8444d92fa3c579"
server_key_version: "encryptionkey@0"
I20260806 10:56:51.286118 13496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "a2291f6829ab46b084f98b6f7e974db4"
format_stamp: "Formatted at 2026-08-06 10:56:51 on dist-test-slave-49h3"
server_key: "3b1505198463f668cac7d66d6d2bebaf"
server_key_iv: "456df9b2734955974a8444d92fa3c579"
server_key_version: "encryptionkey@0"
I20260806 10:56:51.291987 13496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:56:51.296201 13511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:51.297960 13496 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 10:56:51.298174 13496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "a2291f6829ab46b084f98b6f7e974db4"
format_stamp: "Formatted at 2026-08-06 10:56:51 on dist-test-slave-49h3"
server_key: "3b1505198463f668cac7d66d6d2bebaf"
server_key_iv: "456df9b2734955974a8444d92fa3c579"
server_key_version: "encryptionkey@0"
I20260806 10:56:51.298398 13496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:56:51.320667 13496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:56:51.325723 13496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:56:51.326064 13496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:56:51.327144 13496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:56:51.328719 13496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:56:51.328819 13496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:51.328941 13496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:56:51.328994 13496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:51.360795 13496 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:33851
I20260806 10:56:51.360841 13627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:33851 every 8 connection(s)
I20260806 10:56:51.362222 13496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:56:51.366050 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 13496
I20260806 10:56:51.366273 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:56:51.366840 31592 external_mini_cluster.cc:1428] Setting key 113f2f33ae49dc42e0edfc474701c185
Aug 06 10:56:51 dist-test-slave-49h3 krb5kdc[13254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013811, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:56:51.385610 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:51.364858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:34083 (local address 127.30.218.62:42721)
0806 10:56:51.365319 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:51.365324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:51.366549 (+ 1225us) server_negotiation.cc:407] Connection header received
0806 10:56:51.368689 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:51.368694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:51.368781 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:51.368904 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:51.371075 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:51.371831 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:51.374027 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:51.374481 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:51.374590 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:51.379121 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:51.379128 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:51.379131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:51.379175 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:51.381904 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:51.382929 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:51.382933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:51.382935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:51.383076 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:51.383610 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:51.383615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:51.383618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:51.384035 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:51.384197 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:51.384965 (+ 768us) server_negotiation.cc:299] Negotiation successful
0806 10:56:51.385211 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":229,"threads_started":1}
I20260806 10:56:51.386523 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:51.365472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.2:34083)
0806 10:56:51.366350 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:51.366401 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:51.368387 (+ 1986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:51.369193 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:51.369207 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:51.370059 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:56:51.370885 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:51.370913 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:51.372005 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:51.372011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:51.373857 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:51.373873 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:51.374738 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:51.374745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:51.375041 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:51.375823 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:56:51.375859 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:56:51.378776 (+ 2917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:56:51.382106 (+ 3330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:51.382116 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:51.382138 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:51.382808 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:51.383181 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:51.383185 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:51.383189 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:51.383474 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:51.384163 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:56:51.384169 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:56:51.384553 (+ 384us) client_negotiation.cc:769] Sending connection context
0806 10:56:51.384804 (+ 251us) client_negotiation.cc:240] Negotiation successful
0806 10:56:51.385260 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":339,"threads_started":1}
I20260806 10:56:51.388054 13628 heartbeater.cc:347] Connected to a master server at 127.30.218.62:42721
I20260806 10:56:51.388435 13628 heartbeater.cc:464] Registering TS with master...
I20260806 10:56:51.389436 13628 heartbeater.cc:511] Master 127.30.218.62:42721 requested a full tablet report, sending...
I20260806 10:56:51.391245 13299 ts_manager.cc:194] Registered new tserver with Master: a2291f6829ab46b084f98b6f7e974db4 (127.30.218.2:33851)
I20260806 10:56:51.392138 13299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.2@KRBTEST.COM'} at 127.30.218.2:34083
WARNING: no policy specified for kudu/127.30.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
I20260806 10:56:51.440893 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:42721
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:43985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:56:51.793623 13634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:56:51.793941 13634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:56:51.794065 13634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:56:51.801950 13634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:56:51.802078 13634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:56:51.802260 13634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:56:51.812652 13634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:43985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:42721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:56:51.814136 13634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:56:51.815603 13634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:56:51.829325 13642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:51.829603 13639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:51.829386 13640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:51.830461 13634 server_base.cc:1034] running on GCE node
I20260806 10:56:51.831113 13634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:56:51.832167 13634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:56:51.833410 13634 hybrid_clock.cc:648] HybridClock initialized: now 1786013811833355 us; error 44 us; skew 500 ppm
I20260806 10:56:51.833797 13634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:56:51.834256 13490 heartbeater.cc:503] Master 127.30.218.62:42721 was elected leader, sending a full tablet report...
Aug 06 10:56:51 dist-test-slave-49h3 krb5kdc[13254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013811, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:51.837738 13634 init.cc:377] Logged in from keytab as kudu/127.30.218.3@KRBTEST.COM (short username kudu)
I20260806 10:56:51.839386 13634 webserver.cc:460] Webserver started at http://127.30.218.3:43225/ using document root <none> and password file <none>
I20260806 10:56:51.840095 13634 fs_manager.cc:359] Metadata directory not provided
I20260806 10:56:51.840205 13634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:56:51.840471 13634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:56:51.843606 13634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "ae6994110a3b4d93bca7fadcaebafdc5"
format_stamp: "Formatted at 2026-08-06 10:56:51 on dist-test-slave-49h3"
server_key: "25ee91cc2976bb4fb22f1f6fb0dc427e"
server_key_iv: "5fb92550d947966e28c49e76a3ac3f10"
server_key_version: "encryptionkey@0"
I20260806 10:56:51.844420 13634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "ae6994110a3b4d93bca7fadcaebafdc5"
format_stamp: "Formatted at 2026-08-06 10:56:51 on dist-test-slave-49h3"
server_key: "25ee91cc2976bb4fb22f1f6fb0dc427e"
server_key_iv: "5fb92550d947966e28c49e76a3ac3f10"
server_key_version: "encryptionkey@0"
I20260806 10:56:51.850222 13634 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 10:56:51.854279 13649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:51.855796 13634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:56:51.855968 13634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "ae6994110a3b4d93bca7fadcaebafdc5"
format_stamp: "Formatted at 2026-08-06 10:56:51 on dist-test-slave-49h3"
server_key: "25ee91cc2976bb4fb22f1f6fb0dc427e"
server_key_iv: "5fb92550d947966e28c49e76a3ac3f10"
server_key_version: "encryptionkey@0"
I20260806 10:56:51.856166 13634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:56:51.889554 13634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:56:51.893886 13634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:56:51.894157 13634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:56:51.895201 13634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:56:51.896649 13634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:56:51.896726 13634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:51.896826 13634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:56:51.896878 13634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:51.929244 13634 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:41919
I20260806 10:56:51.929291 13762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:41919 every 8 connection(s)
I20260806 10:56:51.930680 13634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:56:51.933791 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 13634
I20260806 10:56:51.933995 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:56:51.934582 31592 external_mini_cluster.cc:1428] Setting key 0fc4bbe6035c9165980535459af66854
Aug 06 10:56:51 dist-test-slave-49h3 krb5kdc[13254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013811, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:56:51.956553 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:51.933358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:60135 (local address 127.30.218.62:42721)
0806 10:56:51.933926 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:51.933932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:51.935375 (+ 1443us) server_negotiation.cc:407] Connection header received
0806 10:56:51.938142 (+ 2767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:51.938148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:51.938238 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:51.938344 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:51.940495 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:51.941697 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:51.944219 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:51.944649 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:51.944768 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:51.949661 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:51.949668 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:51.949673 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:51.949725 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:51.952667 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:51.953587 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:51.953592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:51.953595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:51.953750 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:51.954301 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:51.954306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:51.954308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:51.954775 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:51.954959 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:51.955902 (+ 943us) server_negotiation.cc:299] Negotiation successful
0806 10:56:51.956153 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":268,"threads_started":1}
I20260806 10:56:51.957556 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:51.933925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.3:60135)
0806 10:56:51.934829 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:51.934897 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:51.937693 (+ 2796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:51.938662 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:51.938676 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:51.939457 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:56:51.940274 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:51.940299 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:51.941918 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:51.941924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:51.943973 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:51.943995 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:51.944909 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:51.944917 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:51.945165 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:51.946002 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:56:51.946056 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:56:51.949160 (+ 3104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:56:51.952841 (+ 3681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:51.952851 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:51.952868 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:51.953428 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:51.953863 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:51.953867 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:51.953869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:51.954071 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:51.954922 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:56:51.954929 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:56:51.955426 (+ 497us) client_negotiation.cc:769] Sending connection context
0806 10:56:51.955741 (+ 315us) client_negotiation.cc:240] Negotiation successful
0806 10:56:51.956265 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":333,"threads_started":1}
I20260806 10:56:51.959079 13763 heartbeater.cc:347] Connected to a master server at 127.30.218.62:42721
I20260806 10:56:51.959407 13763 heartbeater.cc:464] Registering TS with master...
I20260806 10:56:51.960379 13763 heartbeater.cc:511] Master 127.30.218.62:42721 requested a full tablet report, sending...
I20260806 10:56:51.962023 13299 ts_manager.cc:194] Registered new tserver with Master: ae6994110a3b4d93bca7fadcaebafdc5 (127.30.218.3:41919)
I20260806 10:56:51.962882 13299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.3@KRBTEST.COM'} at 127.30.218.3:60135
I20260806 10:56:51.969687 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:56:51.989030 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:51.974909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41012 (local address 127.30.218.62:42721)
0806 10:56:51.975134 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:51.975142 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:51.975485 (+ 343us) server_negotiation.cc:407] Connection header received
0806 10:56:51.975679 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:51.975683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:51.975769 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:51.975863 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:51.977586 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:51.978371 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:51.979847 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:51.980283 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:51.980407 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:51.983109 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:51.983115 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:51.983118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:51.983163 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:51.985926 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:51.986791 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:51.986796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:51.986798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:51.986941 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:51.987344 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:51.987348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:51.987350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:51.987739 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:51.987998 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:51.988436 (+ 438us) server_negotiation.cc:299] Negotiation successful
0806 10:56:51.988698 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 10:56:51.993635 13299 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:56:52.000980 13299 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:56:52.036960 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:52.014550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:41919 (local address 127.0.0.1:60508)
0806 10:56:52.025947 (+ 11397us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:52.026046 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:52.026338 (+ 292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:52.028583 (+ 2245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:52.028592 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:52.028643 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:56:52.029232 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:52.029253 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.031380 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.031389 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:52.033863 (+ 2474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:52.033883 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.035841 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.035848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:52.036070 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:52.036130 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 10:56:52.036260 (+ 130us) client_negotiation.cc:240] Negotiation successful
0806 10:56:52.036574 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11000,"thread_start_us":14195,"threads_started":1}
I20260806 10:56:52.036962 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:52.014808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60508 (local address 127.30.218.3:41919)
0806 10:56:52.027192 (+ 12384us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:52.027205 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:52.027248 (+ 43us) server_negotiation.cc:407] Connection header received
0806 10:56:52.027380 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:52.027391 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:52.027721 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:52.027953 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:56:52.029525 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.031191 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.034092 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.035639 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.035770 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:52.036050 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:52.036332 (+ 282us) server_negotiation.cc:299] Negotiation successful
0806 10:56:52.036617 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12139,"thread_start_us":13429,"threads_started":1}
I20260806 10:56:52.039045 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:52.017376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:43791 (local address 127.0.0.1:59276)
0806 10:56:52.029680 (+ 12304us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:52.029698 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:52.029828 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:52.031040 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:52.031045 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:52.031071 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:56:52.031459 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:52.031477 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.032850 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.032856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:52.035341 (+ 2485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:52.035358 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.037758 (+ 2400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.037765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:52.038280 (+ 515us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:52.038316 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 10:56:52.038415 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 10:56:52.038556 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12181,"mutex_wait_us":358,"thread_start_us":11915,"threads_started":1}
I20260806 10:56:52.039108 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:52.014785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:33851 (local address 127.0.0.1:46190)
0806 10:56:52.029062 (+ 14277us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:52.029083 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:52.029347 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:52.030454 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:52.030459 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:52.030488 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:56:52.030874 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:52.030887 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.032800 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.032806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:52.035143 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:52.035157 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.037509 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.037517 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:52.038066 (+ 549us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:52.038419 (+ 353us) client_negotiation.cc:769] Sending connection context
0806 10:56:52.038508 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 10:56:52.038655 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14133,"mutex_wait_us":437,"thread_start_us":10567,"threads_started":1}
I20260806 10:56:52.039754 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:52.018002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59276 (local address 127.30.218.1:43791)
0806 10:56:52.030420 (+ 12418us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:52.030429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:52.030459 (+ 30us) server_negotiation.cc:407] Connection header received
0806 10:56:52.030575 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:52.030583 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:52.030822 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:52.031024 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:56:52.031656 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.032696 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.035539 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.036965 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.037534 (+ 569us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:52.037779 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:52.039334 (+ 1555us) server_negotiation.cc:299] Negotiation successful
0806 10:56:52.039488 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12205,"thread_start_us":12619,"threads_started":1}
I20260806 10:56:52.039994 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:52.015423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46190 (local address 127.30.218.2:33851)
0806 10:56:52.028303 (+ 12880us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:52.028313 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:52.029225 (+ 912us) server_negotiation.cc:407] Connection header received
0806 10:56:52.029640 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:52.029650 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:52.029943 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:52.030132 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:56:52.031071 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.032644 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.035314 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.036928 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.037103 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:52.037324 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:52.039444 (+ 2120us) server_negotiation.cc:299] Negotiation successful
0806 10:56:52.039682 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12670,"thread_start_us":12925,"threads_started":1}
I20260806 10:56:52.041772 13697 tablet_service.cc:1467] Processing CreateTablet for tablet 836b72c38c2142f19bb97ae37f9bb2a0 (DEFAULT_TABLE table=test-table [id=c671758f9cac4800a9a09eb39b104ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:56:52.043223 13697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 836b72c38c2142f19bb97ae37f9bb2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:56:52.043689 13424 tablet_service.cc:1467] Processing CreateTablet for tablet 836b72c38c2142f19bb97ae37f9bb2a0 (DEFAULT_TABLE table=test-table [id=c671758f9cac4800a9a09eb39b104ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:56:52.043687 13560 tablet_service.cc:1467] Processing CreateTablet for tablet 836b72c38c2142f19bb97ae37f9bb2a0 (DEFAULT_TABLE table=test-table [id=c671758f9cac4800a9a09eb39b104ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:56:52.045188 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 836b72c38c2142f19bb97ae37f9bb2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:56:52.045181 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 836b72c38c2142f19bb97ae37f9bb2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:56:52.056375 13782 tablet_bootstrap.cc:492] T 836b72c38c2142f19bb97ae37f9bb2a0 P ae6994110a3b4d93bca7fadcaebafdc5: Bootstrap starting.
I20260806 10:56:52.061232 13782 tablet_bootstrap.cc:654] T 836b72c38c2142f19bb97ae37f9bb2a0 P ae6994110a3b4d93bca7fadcaebafdc5: Neither blocks nor log segments found. Creating new log.
I20260806 10:56:52.062250 13783 tablet_bootstrap.cc:492] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4: Bootstrap starting.
I20260806 10:56:52.062724 13782 log.cc:824] T 836b72c38c2142f19bb97ae37f9bb2a0 P ae6994110a3b4d93bca7fadcaebafdc5: Log is configured to *not* fsync() on all Append() calls
I20260806 10:56:52.063231 13784 tablet_bootstrap.cc:492] T 836b72c38c2142f19bb97ae37f9bb2a0 P 3d2c5277ea7546a3bbdef1d58c93fba2: Bootstrap starting.
I20260806 10:56:52.065920 13782 tablet_bootstrap.cc:492] T 836b72c38c2142f19bb97ae37f9bb2a0 P ae6994110a3b4d93bca7fadcaebafdc5: No bootstrap required, opened a new log
I20260806 10:56:52.066238 13782 ts_tablet_manager.cc:1397] T 836b72c38c2142f19bb97ae37f9bb2a0 P ae6994110a3b4d93bca7fadcaebafdc5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260806 10:56:52.067343 13783 tablet_bootstrap.cc:654] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4: Neither blocks nor log segments found. Creating new log.
I20260806 10:56:52.067657 13784 tablet_bootstrap.cc:654] T 836b72c38c2142f19bb97ae37f9bb2a0 P 3d2c5277ea7546a3bbdef1d58c93fba2: Neither blocks nor log segments found. Creating new log.
I20260806 10:56:52.068850 13784 log.cc:824] T 836b72c38c2142f19bb97ae37f9bb2a0 P 3d2c5277ea7546a3bbdef1d58c93fba2: Log is configured to *not* fsync() on all Append() calls
I20260806 10:56:52.068848 13783 log.cc:824] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4: Log is configured to *not* fsync() on all Append() calls
I20260806 10:56:52.072199 13783 tablet_bootstrap.cc:492] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4: No bootstrap required, opened a new log
I20260806 10:56:52.072336 13784 tablet_bootstrap.cc:492] T 836b72c38c2142f19bb97ae37f9bb2a0 P 3d2c5277ea7546a3bbdef1d58c93fba2: No bootstrap required, opened a new log
I20260806 10:56:52.072531 13783 ts_tablet_manager.cc:1397] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260806 10:56:52.072638 13784 ts_tablet_manager.cc:1397] T 836b72c38c2142f19bb97ae37f9bb2a0 P 3d2c5277ea7546a3bbdef1d58c93fba2: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260806 10:56:52.076788 13782 raft_consensus.cc:348] T 836b72c38c2142f19bb97ae37f9bb2a0 P ae6994110a3b4d93bca7fadcaebafdc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } } peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } }
I20260806 10:56:52.077346 13782 raft_consensus.cc:374] T 836b72c38c2142f19bb97ae37f9bb2a0 P ae6994110a3b4d93bca7fadcaebafdc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:56:52.077495 13782 raft_consensus.cc:729] T 836b72c38c2142f19bb97ae37f9bb2a0 P ae6994110a3b4d93bca7fadcaebafdc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae6994110a3b4d93bca7fadcaebafdc5, State: Initialized, Role: FOLLOWER
I20260806 10:56:52.078387 13782 consensus_queue.cc:260] T 836b72c38c2142f19bb97ae37f9bb2a0 P ae6994110a3b4d93bca7fadcaebafdc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } } peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } }
I20260806 10:56:52.080539 13763 heartbeater.cc:503] Master 127.30.218.62:42721 was elected leader, sending a full tablet report...
I20260806 10:56:52.081418 13782 ts_tablet_manager.cc:1428] T 836b72c38c2142f19bb97ae37f9bb2a0 P ae6994110a3b4d93bca7fadcaebafdc5: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260806 10:56:52.081552 13783 raft_consensus.cc:348] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } } peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } }
I20260806 10:56:52.082085 13783 raft_consensus.cc:374] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:56:52.082204 13783 raft_consensus.cc:729] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2291f6829ab46b084f98b6f7e974db4, State: Initialized, Role: FOLLOWER
I20260806 10:56:52.082948 13783 consensus_queue.cc:260] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } } peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } }
I20260806 10:56:52.083487 13784 raft_consensus.cc:348] T 836b72c38c2142f19bb97ae37f9bb2a0 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } } peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } }
I20260806 10:56:52.083986 13784 raft_consensus.cc:374] T 836b72c38c2142f19bb97ae37f9bb2a0 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:56:52.084343 13628 heartbeater.cc:503] Master 127.30.218.62:42721 was elected leader, sending a full tablet report...
I20260806 10:56:52.085057 13783 ts_tablet_manager.cc:1428] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4: Time spent starting tablet: real 0.012s user 0.008s sys 0.006s
I20260806 10:56:52.085644 13784 raft_consensus.cc:729] T 836b72c38c2142f19bb97ae37f9bb2a0 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d2c5277ea7546a3bbdef1d58c93fba2, State: Initialized, Role: FOLLOWER
I20260806 10:56:52.086437 13784 consensus_queue.cc:260] T 836b72c38c2142f19bb97ae37f9bb2a0 P 3d2c5277ea7546a3bbdef1d58c93fba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } } peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } }
I20260806 10:56:52.090245 13784 ts_tablet_manager.cc:1428] T 836b72c38c2142f19bb97ae37f9bb2a0 P 3d2c5277ea7546a3bbdef1d58c93fba2: Time spent starting tablet: real 0.017s user 0.015s sys 0.002s
W20260806 10:56:52.114385 13629 tablet.cc:2369] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:56:52.135422 13789 raft_consensus.cc:482] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:56:52.135759 13789 raft_consensus.cc:504] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } } peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } }
I20260806 10:56:52.137630 13789 leader_election.cc:288] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae6994110a3b4d93bca7fadcaebafdc5 (127.30.218.3:41919), 3d2c5277ea7546a3bbdef1d58c93fba2 (127.30.218.1:43791)
I20260806 10:56:52.147238 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:52.138253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:60739 (local address 127.30.218.1:43791)
0806 10:56:52.138454 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:52.138459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:52.139276 (+ 817us) server_negotiation.cc:407] Connection header received
0806 10:56:52.139473 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:52.139479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:52.139562 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:52.139670 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:56:52.140449 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.141598 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.144726 (+ 3128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.146043 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.146226 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:52.146307 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:52.146826 (+ 519us) server_negotiation.cc:299] Negotiation successful
0806 10:56:52.146975 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 10:56:52.147238 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:52.138143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:57005 (local address 127.30.218.3:41919)
0806 10:56:52.138373 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:52.138379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:52.139278 (+ 899us) server_negotiation.cc:407] Connection header received
0806 10:56:52.139473 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:52.139479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:52.139562 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:52.139663 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:56:52.140449 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.141598 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.144721 (+ 3123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.146043 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.146226 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:52.146307 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:52.146826 (+ 519us) server_negotiation.cc:299] Negotiation successful
0806 10:56:52.146975 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 10:56:52.147457 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:52.138125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:43791 (local address 127.30.218.2:60739)
0806 10:56:52.139122 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:52.139147 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:52.139327 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:52.139698 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:52.139703 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:52.139736 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:56:52.140326 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:52.140335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.141740 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.141746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:52.144506 (+ 2760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:52.144529 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.146312 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.146319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:52.146587 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:52.146658 (+ 71us) client_negotiation.cc:769] Sending connection context
0806 10:56:52.146759 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 10:56:52.146947 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":286,"spinlock_wait_cycles":70016,"thread_start_us":529,"threads_started":1}
I20260806 10:56:52.147815 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:52.138136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:41919 (local address 127.30.218.2:57005)
0806 10:56:52.139160 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:52.139176 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:52.139327 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:52.139698 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:52.139703 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:52.139744 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:56:52.140256 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:52.140278 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.141743 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.141747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:52.144506 (+ 2759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:52.144529 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.146314 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.146319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:52.146583 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:52.146652 (+ 69us) client_negotiation.cc:769] Sending connection context
0806 10:56:52.146759 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 10:56:52.146947 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":247,"spinlock_wait_cycles":3456,"thread_start_us":467,"threads_started":1}
I20260806 10:56:52.148609 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836b72c38c2142f19bb97ae37f9bb2a0" candidate_uuid: "a2291f6829ab46b084f98b6f7e974db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" is_pre_election: true
I20260806 10:56:52.148793 13718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836b72c38c2142f19bb97ae37f9bb2a0" candidate_uuid: "a2291f6829ab46b084f98b6f7e974db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" is_pre_election: true
I20260806 10:56:52.149299 13718 raft_consensus.cc:2454] T 836b72c38c2142f19bb97ae37f9bb2a0 P ae6994110a3b4d93bca7fadcaebafdc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2291f6829ab46b084f98b6f7e974db4 in term 0.
I20260806 10:56:52.149299 13444 raft_consensus.cc:2454] T 836b72c38c2142f19bb97ae37f9bb2a0 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2291f6829ab46b084f98b6f7e974db4 in term 0.
I20260806 10:56:52.150065 13514 leader_election.cc:302] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2291f6829ab46b084f98b6f7e974db4, ae6994110a3b4d93bca7fadcaebafdc5; no voters:
I20260806 10:56:52.150686 13789 raft_consensus.cc:2781] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:56:52.150828 13789 raft_consensus.cc:482] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:56:52.150933 13789 raft_consensus.cc:3037] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:56:52.153713 13789 raft_consensus.cc:504] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } } peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } }
I20260806 10:56:52.154404 13789 leader_election.cc:288] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4 [CANDIDATE]: Term 1 election: Requested vote from peers ae6994110a3b4d93bca7fadcaebafdc5 (127.30.218.3:41919), 3d2c5277ea7546a3bbdef1d58c93fba2 (127.30.218.1:43791)
I20260806 10:56:52.155066 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836b72c38c2142f19bb97ae37f9bb2a0" candidate_uuid: "a2291f6829ab46b084f98b6f7e974db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2"
I20260806 10:56:52.155064 13718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836b72c38c2142f19bb97ae37f9bb2a0" candidate_uuid: "a2291f6829ab46b084f98b6f7e974db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae6994110a3b4d93bca7fadcaebafdc5"
I20260806 10:56:52.155311 13444 raft_consensus.cc:3037] T 836b72c38c2142f19bb97ae37f9bb2a0 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:56:52.155313 13718 raft_consensus.cc:3037] T 836b72c38c2142f19bb97ae37f9bb2a0 P ae6994110a3b4d93bca7fadcaebafdc5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:56:52.158762 13444 raft_consensus.cc:2454] T 836b72c38c2142f19bb97ae37f9bb2a0 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2291f6829ab46b084f98b6f7e974db4 in term 1.
I20260806 10:56:52.158762 13718 raft_consensus.cc:2454] T 836b72c38c2142f19bb97ae37f9bb2a0 P ae6994110a3b4d93bca7fadcaebafdc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2291f6829ab46b084f98b6f7e974db4 in term 1.
I20260806 10:56:52.159363 13515 leader_election.cc:302] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3d2c5277ea7546a3bbdef1d58c93fba2, a2291f6829ab46b084f98b6f7e974db4, ae6994110a3b4d93bca7fadcaebafdc5; no voters:
I20260806 10:56:52.159791 13789 raft_consensus.cc:2781] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:56:52.160275 13789 raft_consensus.cc:686] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4 [term 1 LEADER]: Becoming Leader. State: Replica: a2291f6829ab46b084f98b6f7e974db4, State: Running, Role: LEADER
I20260806 10:56:52.160861 13789 consensus_queue.cc:237] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } } peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } }
I20260806 10:56:52.165602 13299 catalog_manager.cc:5495] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2291f6829ab46b084f98b6f7e974db4 (127.30.218.2). New cstate: current_term: 1 leader_uuid: "a2291f6829ab46b084f98b6f7e974db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } health_report { overall_health: UNKNOWN } } }
W20260806 10:56:52.182416 13764 tablet.cc:2369] T 836b72c38c2142f19bb97ae37f9bb2a0 P ae6994110a3b4d93bca7fadcaebafdc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:56:52.196898 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:52.191273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46200 (local address 127.30.218.2:33851)
0806 10:56:52.191473 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:52.191479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:52.191501 (+ 22us) server_negotiation.cc:407] Connection header received
0806 10:56:52.191749 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:52.191754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:52.191837 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:52.191976 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:56:52.192373 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.193093 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.194668 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.195090 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.195218 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:52.195760 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:56:52.196292 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:56:52.196442 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:52.196523 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 10:56:52.196617 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 10:56:52.205986 13444 raft_consensus.cc:1261] T 836b72c38c2142f19bb97ae37f9bb2a0 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Refusing update from remote peer a2291f6829ab46b084f98b6f7e974db4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:56:52.205984 13718 raft_consensus.cc:1261] T 836b72c38c2142f19bb97ae37f9bb2a0 P ae6994110a3b4d93bca7fadcaebafdc5 [term 1 FOLLOWER]: Refusing update from remote peer a2291f6829ab46b084f98b6f7e974db4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:56:52.207083 13793 consensus_queue.cc:1035] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:56:52.207494 13789 consensus_queue.cc:1035] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:56:52.220615 13796 mvcc.cc:205] Tried to move back new op lower bound from 7315512574784434176 to 7315512574615175168. Current Snapshot: MvccSnapshot[applied={T|T < 7315512574784434176}]
I20260806 10:56:52.236902 13299 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:56:52.237190 13299 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:56:52.241637 13299 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67: Sending DeleteTablet for 3 replicas of tablet 836b72c38c2142f19bb97ae37f9bb2a0
I20260806 10:56:52.243093 13697 tablet_service.cc:1514] Processing DeleteTablet for tablet 836b72c38c2142f19bb97ae37f9bb2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:56:52 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:60508
I20260806 10:56:52.243124 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet 836b72c38c2142f19bb97ae37f9bb2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:56:52 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:46190
I20260806 10:56:52.243574 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet 836b72c38c2142f19bb97ae37f9bb2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:56:52 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:59276
I20260806 10:56:52.244812 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 13361
I20260806 10:56:52.252763 13807 tablet_replica.cc:330] T 836b72c38c2142f19bb97ae37f9bb2a0 P 3d2c5277ea7546a3bbdef1d58c93fba2: stopping tablet replica
I20260806 10:56:52.253404 13807 raft_consensus.cc:2229] T 836b72c38c2142f19bb97ae37f9bb2a0 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:56:52.254457 13807 raft_consensus.cc:2258] T 836b72c38c2142f19bb97ae37f9bb2a0 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:56:52.262023 13807 ts_tablet_manager.cc:1905] T 836b72c38c2142f19bb97ae37f9bb2a0 P 3d2c5277ea7546a3bbdef1d58c93fba2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:56:52.267966 13806 tablet_replica.cc:330] T 836b72c38c2142f19bb97ae37f9bb2a0 P ae6994110a3b4d93bca7fadcaebafdc5: stopping tablet replica
I20260806 10:56:52.268647 13805 tablet_replica.cc:330] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4: stopping tablet replica
I20260806 10:56:52.269492 13806 raft_consensus.cc:2229] T 836b72c38c2142f19bb97ae37f9bb2a0 P ae6994110a3b4d93bca7fadcaebafdc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:56:52.269975 13805 raft_consensus.cc:2229] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:56:52.270659 13806 raft_consensus.cc:2258] T 836b72c38c2142f19bb97ae37f9bb2a0 P ae6994110a3b4d93bca7fadcaebafdc5 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 06 10:56:52 dist-test-slave-49h3 krb5kdc[13254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.1@KRBTEST.COM
I20260806 10:56:52.272173 13805 raft_consensus.cc:2258] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:56:52.272410 13807 ts_tablet_manager.cc:1918] T 836b72c38c2142f19bb97ae37f9bb2a0 P 3d2c5277ea7546a3bbdef1d58c93fba2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:56:52.272646 13807 log.cc:1196] T 836b72c38c2142f19bb97ae37f9bb2a0 P 3d2c5277ea7546a3bbdef1d58c93fba2: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/wal/wals/836b72c38c2142f19bb97ae37f9bb2a0
I20260806 10:56:52.273303 13807 ts_tablet_manager.cc:1939] T 836b72c38c2142f19bb97ae37f9bb2a0 P 3d2c5277ea7546a3bbdef1d58c93fba2: Deleting consensus metadata
I20260806 10:56:52.275471 13286 catalog_manager.cc:4841] TS 3d2c5277ea7546a3bbdef1d58c93fba2 (127.30.218.1:43791): tablet 836b72c38c2142f19bb97ae37f9bb2a0 (table test-table [id=c671758f9cac4800a9a09eb39b104ed0]) successfully deleted
I20260806 10:56:52.277563 13806 ts_tablet_manager.cc:1905] T 836b72c38c2142f19bb97ae37f9bb2a0 P ae6994110a3b4d93bca7fadcaebafdc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:56:52.278034 13805 ts_tablet_manager.cc:1905] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:56:52.280654 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:52.245779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59282 (local address 127.30.218.1:43791)
0806 10:56:52.246000 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:52.246007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:52.259295 (+ 13288us) server_negotiation.cc:407] Connection header received
0806 10:56:52.259928 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:52.259934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:52.260029 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:52.260116 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:52.262631 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.263784 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.265838 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.266465 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.266575 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:52.272569 (+ 5994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:52.272592 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:52.272610 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:52.272676 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:52.276718 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:52.277832 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:52.277842 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:52.277845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:52.278027 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:52.278545 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:52.278550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:52.278553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:52.279050 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:52.279356 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:52.279993 (+ 637us) server_negotiation.cc:299] Negotiation successful
0806 10:56:52.280275 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 10:56:52.282167 13485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:56:52.284976 13806 ts_tablet_manager.cc:1918] T 836b72c38c2142f19bb97ae37f9bb2a0 P ae6994110a3b4d93bca7fadcaebafdc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:56:52.285185 13806 log.cc:1196] T 836b72c38c2142f19bb97ae37f9bb2a0 P ae6994110a3b4d93bca7fadcaebafdc5: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/836b72c38c2142f19bb97ae37f9bb2a0
I20260806 10:56:52.285802 13806 ts_tablet_manager.cc:1939] T 836b72c38c2142f19bb97ae37f9bb2a0 P ae6994110a3b4d93bca7fadcaebafdc5: Deleting consensus metadata
I20260806 10:56:52.287918 13285 catalog_manager.cc:4841] TS ae6994110a3b4d93bca7fadcaebafdc5 (127.30.218.3:41919): tablet 836b72c38c2142f19bb97ae37f9bb2a0 (table test-table [id=c671758f9cac4800a9a09eb39b104ed0]) successfully deleted
I20260806 10:56:52.288552 13805 ts_tablet_manager.cc:1918] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:56:52.288822 13805 log.cc:1196] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/836b72c38c2142f19bb97ae37f9bb2a0
I20260806 10:56:52.289492 13805 ts_tablet_manager.cc:1939] T 836b72c38c2142f19bb97ae37f9bb2a0 P a2291f6829ab46b084f98b6f7e974db4: Deleting consensus metadata
I20260806 10:56:52.291486 13283 catalog_manager.cc:4841] TS a2291f6829ab46b084f98b6f7e974db4 (127.30.218.2:33851): tablet 836b72c38c2142f19bb97ae37f9bb2a0 (table test-table [id=c671758f9cac4800a9a09eb39b104ed0]) successfully deleted
I20260806 10:56:52.400789 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 13361
I20260806 10:56:52.419977 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 13496
Aug 06 10:56:52 dist-test-slave-49h3 krb5kdc[13254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.2@KRBTEST.COM
I20260806 10:56:52.437726 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:52.421398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46204 (local address 127.30.218.2:33851)
0806 10:56:52.421615 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:52.421620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:52.421637 (+ 17us) server_negotiation.cc:407] Connection header received
0806 10:56:52.421692 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:52.421696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:52.421784 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:52.421886 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:52.423251 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.424099 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.425592 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.426054 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.426189 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:52.430997 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:52.431016 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:52.431033 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:52.431084 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:52.434432 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:52.435336 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:52.435346 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:52.435348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:52.435532 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:52.435975 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:52.435980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:52.435982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:52.436414 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:52.436765 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:52.437057 (+ 292us) server_negotiation.cc:299] Negotiation successful
0806 10:56:52.437322 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260806 10:56:52.438453 13622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:56:52.558008 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 13496
I20260806 10:56:52.576508 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 13634
Aug 06 10:56:52 dist-test-slave-49h3 krb5kdc[13254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.3@KRBTEST.COM
I20260806 10:56:52.592852 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:52.577284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.30.218.3:41919)
0806 10:56:52.577490 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:52.577496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:52.577517 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:56:52.577721 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:52.577727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:52.577820 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:52.577919 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:52.579238 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.579981 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.581302 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:52.581959 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:52.582128 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:52.586826 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:52.586847 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:52.586864 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:52.586908 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:52.589963 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:52.590813 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:52.590820 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:52.590823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:52.590986 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:52.591298 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:52.591301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:52.591303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:52.591677 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:52.591919 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:52.592363 (+ 444us) server_negotiation.cc:299] Negotiation successful
0806 10:56:52.592570 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 10:56:52.593679 13758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:56:52.686962 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 13634
I20260806 10:56:52.704782 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 13267
I20260806 10:56:52.706166 13330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:56:52.822151 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 13267
I20260806 10:56:52.846186 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:42721
--webserver_interface=127.30.218.62
--webserver_port=41045
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:43985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:42721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:56:53.205587 13813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:56:53.205942 13813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:56:53.206118 13813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:56:53.213990 13813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:56:53.214143 13813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:56:53.214203 13813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:56:53.214287 13813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:56:53.214340 13813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:56:53.225077 13813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:43985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:42721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:42721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=41045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:56:53.226641 13813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:56:53.228178 13813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:56:53.254467 13822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:53.254380 13819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:53.254400 13820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:53.255388 13813 server_base.cc:1034] running on GCE node
I20260806 10:56:53.256345 13813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:56:53.258230 13813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:56:53.259495 13813 hybrid_clock.cc:648] HybridClock initialized: now 1786013813259438 us; error 49 us; skew 500 ppm
I20260806 10:56:53.259954 13813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:56:53 dist-test-slave-49h3 krb5kdc[13254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013813, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:53.267302 13813 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:56:53.269304 13813 webserver.cc:460] Webserver started at http://127.30.218.62:41045/ using document root <none> and password file <none>
I20260806 10:56:53.270210 13813 fs_manager.cc:359] Metadata directory not provided
I20260806 10:56:53.270375 13813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:56:53.277132 13813 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 10:56:53.280469 13829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:53.282132 13813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 10:56:53.282333 13813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "1a866dfcd2fd4db3a6082469f1e47e67"
format_stamp: "Formatted at 2026-08-06 10:56:50 on dist-test-slave-49h3"
server_key: "57b918a3d7eeb3c385415ab2b4b7269e"
server_key_iv: "6d4b4e9646b87a0dd2e935c73a20c37e"
server_key_version: "encryptionkey@0"
I20260806 10:56:53.283501 13813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:56:53.314931 13813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:56:53.319907 13813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:56:53.320212 13813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:56:53.339962 13813 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:42721
I20260806 10:56:53.339986 13883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:42721 every 8 connection(s)
I20260806 10:56:53.341717 13813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:56:53.349257 13884 sys_catalog.cc:263] Verifying existing consensus state
I20260806 10:56:53.350294 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 13813
I20260806 10:56:53.351346 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:43791
--local_ip_for_outbound_sockets=127.30.218.1
--tserver_master_addrs=127.30.218.62:42721
--webserver_port=39225
--webserver_interface=127.30.218.1
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:43985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:56:53.354404 13884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67: Bootstrap starting.
I20260806 10:56:53.378010 13884 log.cc:824] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67: Log is configured to *not* fsync() on all Append() calls
I20260806 10:56:53.392438 13884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 10:56:53.393040 13884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67: Bootstrap complete.
I20260806 10:56:53.402860 13884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a866dfcd2fd4db3a6082469f1e47e67" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42721 } }
I20260806 10:56:53.403802 13884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a866dfcd2fd4db3a6082469f1e47e67, State: Initialized, Role: FOLLOWER
I20260806 10:56:53.404489 13884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a866dfcd2fd4db3a6082469f1e47e67" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42721 } }
I20260806 10:56:53.404757 13884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:56:53.404889 13884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:56:53.405057 13884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 1 FOLLOWER]: Advancing to term 2
I20260806 10:56:53.409423 13884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a866dfcd2fd4db3a6082469f1e47e67" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42721 } }
I20260806 10:56:53.410151 13884 leader_election.cc:302] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a866dfcd2fd4db3a6082469f1e47e67; no voters:
I20260806 10:56:53.410801 13884 leader_election.cc:288] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 10:56:53.410954 13888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 10:56:53.412348 13888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 2 LEADER]: Becoming Leader. State: Replica: 1a866dfcd2fd4db3a6082469f1e47e67, State: Running, Role: LEADER
I20260806 10:56:53.413036 13888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a866dfcd2fd4db3a6082469f1e47e67" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42721 } }
I20260806 10:56:53.413517 13884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:56:53.415822 13889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1a866dfcd2fd4db3a6082469f1e47e67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a866dfcd2fd4db3a6082469f1e47e67" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42721 } } }
I20260806 10:56:53.416308 13889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [sys.catalog]: This master's current role is: LEADER
I20260806 10:56:53.416879 13890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a866dfcd2fd4db3a6082469f1e47e67. Latest consensus state: current_term: 2 leader_uuid: "1a866dfcd2fd4db3a6082469f1e47e67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a866dfcd2fd4db3a6082469f1e47e67" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42721 } } }
I20260806 10:56:53.417306 13890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [sys.catalog]: This master's current role is: LEADER
I20260806 10:56:53.419575 13893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:56:53.429853 13893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:56:53.431861 13893 catalog_manager.cc:1259] Loaded cluster ID: 403cc141659e4e7bb759a4497f8007db
I20260806 10:56:53.432101 13893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:56:53.440414 13893 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 10:56:53.459342 13893 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 10:56:53.477789 13893 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:56:53.479931 13893 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67: Loaded TSK: 0
I20260806 10:56:53.482156 13893 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 10:56:53.738783 13886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:56:53.739161 13886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:56:53.739315 13886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:56:53.747143 13886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:56:53.747282 13886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:56:53.747432 13886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:56:53.758535 13886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:43985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:43791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=39225
--tserver_master_addrs=127.30.218.62:42721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:56:53.759979 13886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:56:53.761371 13886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:56:53.777110 13914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:53.777096 13912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:53.777683 13911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:53.781500 13886 server_base.cc:1034] running on GCE node
I20260806 10:56:53.782467 13886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:56:53.783903 13886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:56:53.785181 13886 hybrid_clock.cc:648] HybridClock initialized: now 1786013813785112 us; error 57 us; skew 500 ppm
I20260806 10:56:53.785626 13886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:56:53 dist-test-slave-49h3 krb5kdc[13254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013813, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:53.790086 13886 init.cc:377] Logged in from keytab as kudu/127.30.218.1@KRBTEST.COM (short username kudu)
I20260806 10:56:53.792273 13886 webserver.cc:460] Webserver started at http://127.30.218.1:39225/ using document root <none> and password file <none>
I20260806 10:56:53.793350 13886 fs_manager.cc:359] Metadata directory not provided
I20260806 10:56:53.793543 13886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:56:53.801249 13886 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 10:56:53.804926 13921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:53.806767 13886 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 10:56:53.806974 13886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "3d2c5277ea7546a3bbdef1d58c93fba2"
format_stamp: "Formatted at 2026-08-06 10:56:50 on dist-test-slave-49h3"
server_key: "7d11dbb0738e614a2868f360eafdc805"
server_key_iv: "f6d5a7dd3dc78cf870bf802703579218"
server_key_version: "encryptionkey@0"
I20260806 10:56:53.808444 13886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:56:53.833173 13886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:56:53.837790 13886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:56:53.838052 13886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:56:53.839135 13886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:56:53.840575 13886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:56:53.840675 13886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:53.840767 13886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:56:53.840821 13886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:53.876745 13886 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:43791
I20260806 10:56:53.876820 14037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:43791 every 8 connection(s)
I20260806 10:56:53.878953 13886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:56:53.888017 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 13886
I20260806 10:56:53.888902 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:33851
--local_ip_for_outbound_sockets=127.30.218.2
--tserver_master_addrs=127.30.218.62:42721
--webserver_port=35963
--webserver_interface=127.30.218.2
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:43985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 10:56:53 dist-test-slave-49h3 krb5kdc[13254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013813, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:56:53.910732 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:53.883464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.1:33183)
0806 10:56:53.884420 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:53.884481 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:53.886918 (+ 2437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:53.889038 (+ 2120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:53.889050 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:53.889950 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:56:53.890766 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:53.890795 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:53.893338 (+ 2543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:53.893342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:53.895333 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:53.895355 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:53.896923 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:53.896931 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:53.897224 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:53.897935 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:56:53.897967 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:56:53.900308 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:56:53.905091 (+ 4783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:53.905101 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:53.905119 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:53.905826 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:53.906460 (+ 634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:53.906465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:53.906469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:53.906735 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:53.907895 (+ 1160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:56:53.907904 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:56:53.908321 (+ 417us) client_negotiation.cc:769] Sending connection context
0806 10:56:53.908624 (+ 303us) client_negotiation.cc:240] Negotiation successful
0806 10:56:53.909207 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":362,"threads_started":1}
I20260806 10:56:53.911789 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:53.883141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:33183 (local address 127.30.218.62:42721)
0806 10:56:53.883794 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:53.883807 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:53.884663 (+ 856us) server_negotiation.cc:407] Connection header received
0806 10:56:53.887680 (+ 3017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:53.887702 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:53.888292 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:53.888823 (+ 531us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:53.891119 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:53.893115 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:53.895563 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:53.896627 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:53.896845 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:53.900668 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:53.900677 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:53.900694 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:53.900744 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:53.904836 (+ 4092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:53.906052 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:53.906062 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:53.906070 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:53.906310 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:53.906869 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:53.906874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:53.906877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:53.907575 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:53.907811 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:53.909887 (+ 2076us) server_negotiation.cc:299] Negotiation successful
0806 10:56:53.910427 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":278,"threads_started":1}
I20260806 10:56:53.915457 14038 heartbeater.cc:347] Connected to a master server at 127.30.218.62:42721
I20260806 10:56:53.915825 14038 heartbeater.cc:464] Registering TS with master...
I20260806 10:56:53.916765 14038 heartbeater.cc:511] Master 127.30.218.62:42721 requested a full tablet report, sending...
I20260806 10:56:53.919440 13848 ts_manager.cc:194] Registered new tserver with Master: 3d2c5277ea7546a3bbdef1d58c93fba2 (127.30.218.1:43791)
I20260806 10:56:53.920997 13848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.1@KRBTEST.COM'} at 127.30.218.1:33183
W20260806 10:56:54.264319 14042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:56:54.264662 14042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:56:54.264791 14042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:56:54.272594 14042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:56:54.272733 14042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:56:54.272890 14042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:56:54.283545 14042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:43985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:33851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=35963
--tserver_master_addrs=127.30.218.62:42721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:56:54.285053 14042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:56:54.286602 14042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:56:54.300688 14050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:54.300726 14048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:54.300748 14047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:54.301376 14042 server_base.cc:1034] running on GCE node
I20260806 10:56:54.302103 14042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:56:54.303145 14042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:56:54.304360 14042 hybrid_clock.cc:648] HybridClock initialized: now 1786013814304307 us; error 38 us; skew 500 ppm
I20260806 10:56:54.304709 14042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:56:54 dist-test-slave-49h3 krb5kdc[13254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013814, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:54.308118 14042 init.cc:377] Logged in from keytab as kudu/127.30.218.2@KRBTEST.COM (short username kudu)
I20260806 10:56:54.309785 14042 webserver.cc:460] Webserver started at http://127.30.218.2:35963/ using document root <none> and password file <none>
I20260806 10:56:54.310515 14042 fs_manager.cc:359] Metadata directory not provided
I20260806 10:56:54.310658 14042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:56:54.316668 14042 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 10:56:54.319777 14057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:54.321247 14042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:56:54.321422 14042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "a2291f6829ab46b084f98b6f7e974db4"
format_stamp: "Formatted at 2026-08-06 10:56:51 on dist-test-slave-49h3"
server_key: "3b1505198463f668cac7d66d6d2bebaf"
server_key_iv: "456df9b2734955974a8444d92fa3c579"
server_key_version: "encryptionkey@0"
I20260806 10:56:54.322588 14042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:56:54.340083 14042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:56:54.344813 14042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:56:54.345117 14042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:56:54.346313 14042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:56:54.347801 14042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:56:54.347916 14042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:54.348023 14042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:56:54.348098 14042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:54.381153 14042 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:33851
I20260806 10:56:54.381217 14175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:33851 every 8 connection(s)
I20260806 10:56:54.382795 14042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:56:54.389818 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 14042
I20260806 10:56:54.390609 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:41919
--local_ip_for_outbound_sockets=127.30.218.3
--tserver_master_addrs=127.30.218.62:42721
--webserver_port=43225
--webserver_interface=127.30.218.3
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:43985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 10:56:54 dist-test-slave-49h3 krb5kdc[13254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013814, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:56:54.405108 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:54.385516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:38775 (local address 127.30.218.62:42721)
0806 10:56:54.385698 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:54.385703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:54.386932 (+ 1229us) server_negotiation.cc:407] Connection header received
0806 10:56:54.388935 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:54.388940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:54.389059 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:54.389172 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:54.391381 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:54.392119 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:54.394377 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:54.394805 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:54.394906 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:54.399136 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:54.399142 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:54.399145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:54.399184 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:54.401839 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:54.402662 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:54.402667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:54.402669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:54.402818 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:54.403274 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:54.403279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:54.403281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:54.403575 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:54.403714 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:54.404478 (+ 764us) server_negotiation.cc:299] Negotiation successful
0806 10:56:54.404754 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 10:56:54.406212 14178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:54.386003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.2:38775)
0806 10:56:54.386756 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:54.386802 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:54.388622 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:54.389495 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:54.389509 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:54.390308 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:56:54.391176 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:54.391203 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:54.392305 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:54.392311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:54.394216 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:54.394235 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:54.395053 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:54.395059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:54.395357 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:54.396127 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:56:54.396166 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:56:54.398850 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:56:54.402023 (+ 3173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:54.402032 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:54.402049 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:54.402518 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:54.402934 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:54.402939 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:54.402943 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:54.403155 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:54.403704 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:56:54.403712 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:56:54.404073 (+ 361us) client_negotiation.cc:769] Sending connection context
0806 10:56:54.404330 (+ 257us) client_negotiation.cc:240] Negotiation successful
0806 10:56:54.404948 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":286,"threads_started":1}
I20260806 10:56:54.407884 14176 heartbeater.cc:347] Connected to a master server at 127.30.218.62:42721
I20260806 10:56:54.408197 14176 heartbeater.cc:464] Registering TS with master...
I20260806 10:56:54.408979 14176 heartbeater.cc:511] Master 127.30.218.62:42721 requested a full tablet report, sending...
I20260806 10:56:54.410714 13848 ts_manager.cc:194] Registered new tserver with Master: a2291f6829ab46b084f98b6f7e974db4 (127.30.218.2:33851)
I20260806 10:56:54.411578 13848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.2@KRBTEST.COM'} at 127.30.218.2:38775
W20260806 10:56:54.747818 14179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:56:54.748188 14179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:56:54.748334 14179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:56:54.756024 14179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:56:54.756198 14179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:56:54.756422 14179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:56:54.766906 14179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:43985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:41919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=43225
--tserver_master_addrs=127.30.218.62:42721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:56:54.768356 14179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:56:54.769781 14179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:56:54.783880 14184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:54.783846 14188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:54.784039 14185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:54.785302 14179 server_base.cc:1034] running on GCE node
I20260806 10:56:54.786082 14179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:56:54.787155 14179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:56:54.788403 14179 hybrid_clock.cc:648] HybridClock initialized: now 1786013814788316 us; error 73 us; skew 500 ppm
I20260806 10:56:54.788760 14179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:56:54 dist-test-slave-49h3 krb5kdc[13254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013814, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:54.792238 14179 init.cc:377] Logged in from keytab as kudu/127.30.218.3@KRBTEST.COM (short username kudu)
I20260806 10:56:54.794013 14179 webserver.cc:460] Webserver started at http://127.30.218.3:43225/ using document root <none> and password file <none>
I20260806 10:56:54.794809 14179 fs_manager.cc:359] Metadata directory not provided
I20260806 10:56:54.794948 14179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:56:54.801096 14179 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 10:56:54.804325 14195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:54.805913 14179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 10:56:54.806150 14179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "ae6994110a3b4d93bca7fadcaebafdc5"
format_stamp: "Formatted at 2026-08-06 10:56:51 on dist-test-slave-49h3"
server_key: "25ee91cc2976bb4fb22f1f6fb0dc427e"
server_key_iv: "5fb92550d947966e28c49e76a3ac3f10"
server_key_version: "encryptionkey@0"
I20260806 10:56:54.807252 14179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:56:54.834734 14179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:56:54.839936 14179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:56:54.840262 14179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:56:54.841509 14179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:56:54.843106 14179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:56:54.843220 14179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:54.843341 14179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:56:54.843432 14179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:54.880915 14179 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:41919
I20260806 10:56:54.880963 14311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:41919 every 8 connection(s)
I20260806 10:56:54.882499 14179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:56:54.884706 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 14179
Aug 06 10:56:54 dist-test-slave-49h3 krb5kdc[13254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013814, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:56:54.901026 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:54.885799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.30.218.62:42721)
0806 10:56:54.886336 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:54.886342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:54.886698 (+ 356us) server_negotiation.cc:407] Connection header received
0806 10:56:54.886889 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:54.886893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:54.886968 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:54.887080 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:54.888976 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:54.889828 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:54.891263 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:54.891816 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:54.891936 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:54.895057 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:54.895063 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:54.895067 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:54.895105 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:54.897888 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:54.898695 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:54.898699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:54.898702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:54.898846 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:54.899355 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:54.899359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:54.899361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:54.899719 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:54.899955 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:54.900399 (+ 444us) server_negotiation.cc:299] Negotiation successful
0806 10:56:54.900657 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":429,"thread_start_us":284,"threads_started":1}
I20260806 10:56:54.905220 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:54.885357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:37689 (local address 127.30.218.62:42721)
0806 10:56:54.885615 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:54.885622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:54.887230 (+ 1608us) server_negotiation.cc:407] Connection header received
0806 10:56:54.889383 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:54.889389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:54.889506 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:54.889599 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:54.891259 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:54.892120 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:54.893986 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:54.894450 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:54.894566 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:54.898400 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:54.898406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:54.898409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:54.898444 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:54.901599 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:54.902689 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:54.902695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:54.902699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:54.902854 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:54.903345 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:54.903350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:54.903353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:54.903755 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:54.903899 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:54.904758 (+ 859us) server_negotiation.cc:299] Negotiation successful
0806 10:56:54.904951 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 10:56:54.906306 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:54.886141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.3:37689)
0806 10:56:54.887051 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:54.887105 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:54.889085 (+ 1980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:54.889881 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:54.889890 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:54.890447 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:56:54.891071 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:54.891090 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:54.892297 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:54.892301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:54.893825 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:54.893842 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:54.894706 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:54.894714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:54.894972 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:54.895663 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:56:54.895697 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:56:54.898153 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:56:54.901760 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:54.901769 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:54.901787 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:54.902504 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:54.902985 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:54.902989 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:54.902991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:54.903219 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:54.903911 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:56:54.903921 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:56:54.904369 (+ 448us) client_negotiation.cc:769] Sending connection context
0806 10:56:54.904630 (+ 261us) client_negotiation.cc:240] Negotiation successful
0806 10:56:54.905073 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":356,"threads_started":1}
I20260806 10:56:54.907781 14312 heartbeater.cc:347] Connected to a master server at 127.30.218.62:42721
I20260806 10:56:54.908105 14312 heartbeater.cc:464] Registering TS with master...
I20260806 10:56:54.908866 14312 heartbeater.cc:511] Master 127.30.218.62:42721 requested a full tablet report, sending...
I20260806 10:56:54.910609 13848 ts_manager.cc:194] Registered new tserver with Master: ae6994110a3b4d93bca7fadcaebafdc5 (127.30.218.3:41919)
I20260806 10:56:54.911526 13848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.3@KRBTEST.COM'} at 127.30.218.3:37689
I20260806 10:56:54.917524 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:56:54.925060 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:54.918690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41034 (local address 127.30.218.62:42721)
0806 10:56:54.918913 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:54.918920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:54.919294 (+ 374us) server_negotiation.cc:407] Connection header received
0806 10:56:54.919542 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:54.919547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:54.919728 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:54.919892 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:56:54.920374 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:54.921199 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:54.922741 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:54.923177 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:54.923291 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:54.923898 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:56:54.924430 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:56:54.924573 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:54.924687 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 10:56:54.924791 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 10:56:54.925108 14038 heartbeater.cc:503] Master 127.30.218.62:42721 was elected leader, sending a full tablet report...
I20260806 10:56:54.926294 13848 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:56:54.934235 13848 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:56:54.962383 14325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:54.948373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:43791 (local address 127.0.0.1:59292)
0806 10:56:54.949550 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:54.949613 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:54.949810 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:54.951964 (+ 2154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:54.951974 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:54.952014 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:56:54.952473 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:54.952487 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:54.955696 (+ 3209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:54.955704 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:54.958563 (+ 2859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:54.958593 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:54.961286 (+ 2693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:54.961295 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:54.961569 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:54.961654 (+ 85us) client_negotiation.cc:769] Sending connection context
0806 10:56:54.961795 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 10:56:54.962015 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"thread_start_us":661,"threads_started":1}
I20260806 10:56:54.962944 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:54.948961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59292 (local address 127.30.218.1:43791)
0806 10:56:54.951216 (+ 2255us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:54.951228 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:54.951264 (+ 36us) server_negotiation.cc:407] Connection header received
0806 10:56:54.951375 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:54.951385 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:54.951686 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:54.952722 (+ 1036us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:56:54.953014 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:54.954719 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:54.958891 (+ 4172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:54.960955 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:54.961746 (+ 791us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:54.962015 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:54.962094 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 10:56:54.962294 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2031,"thread_start_us":2410,"threads_started":1}
I20260806 10:56:54.967336 13970 tablet_service.cc:1467] Processing CreateTablet for tablet 1e65f12167a346a39018672ca5370d74 (DEFAULT_TABLE table=test-table [id=ec940585565a4d28bd545ec703448e70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:56:54.968827 13970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e65f12167a346a39018672ca5370d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:56:54.969074 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:54.950884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:33851 (local address 127.0.0.1:46210)
0806 10:56:54.960167 (+ 9283us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:54.960187 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:54.960318 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:54.962746 (+ 2428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:54.962752 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:54.962785 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:56:54.963175 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:54.963187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:54.964902 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:54.964907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:54.966281 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:54.966294 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:54.968139 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:54.968145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:54.968338 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:54.968372 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 10:56:54.968463 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 10:56:54.968606 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9121,"thread_start_us":8963,"threads_started":1}
I20260806 10:56:54.970065 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:54.951538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46210 (local address 127.30.218.2:33851)
0806 10:56:54.961723 (+ 10185us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:54.961734 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:54.961768 (+ 34us) server_negotiation.cc:407] Connection header received
0806 10:56:54.961882 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:54.961892 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:54.962247 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:54.962478 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:56:54.963336 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:54.964727 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:54.966442 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:54.967845 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:54.967973 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:54.969482 (+ 1509us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:54.969560 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 10:56:54.969735 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9996,"thread_start_us":9824,"threads_started":1}
I20260806 10:56:54.971122 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:54.948532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:41919 (local address 127.0.0.1:60532)
0806 10:56:54.955341 (+ 6809us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:54.955365 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:54.955511 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:54.963976 (+ 8465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:54.963981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:54.964010 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:56:54.964411 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:54.964426 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:54.966667 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:54.966672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:54.968601 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:54.968614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:54.970405 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:54.970412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:54.970592 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:54.970625 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 10:56:54.970689 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 10:56:54.970822 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6654,"thread_start_us":6499,"threads_started":1}
I20260806 10:56:54.972149 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:54.949430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60532 (local address 127.30.218.3:41919)
0806 10:56:54.963364 (+ 13934us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:54.963372 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:54.963406 (+ 34us) server_negotiation.cc:407] Connection header received
0806 10:56:54.963502 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:54.963509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:54.963759 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:54.964644 (+ 885us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:56:54.964943 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:54.966534 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:54.968786 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:54.970170 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:54.971107 (+ 937us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:54.971501 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:54.971571 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 10:56:54.971744 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13753,"thread_start_us":13564,"threads_started":1}
I20260806 10:56:54.973696 14107 tablet_service.cc:1467] Processing CreateTablet for tablet 1e65f12167a346a39018672ca5370d74 (DEFAULT_TABLE table=test-table [id=ec940585565a4d28bd545ec703448e70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:56:54.974922 14107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e65f12167a346a39018672ca5370d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:56:54.975574 14244 tablet_service.cc:1467] Processing CreateTablet for tablet 1e65f12167a346a39018672ca5370d74 (DEFAULT_TABLE table=test-table [id=ec940585565a4d28bd545ec703448e70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:56:54.976903 14244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e65f12167a346a39018672ca5370d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:56:54.984508 14332 tablet_bootstrap.cc:492] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2: Bootstrap starting.
I20260806 10:56:54.988418 14332 tablet_bootstrap.cc:654] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2: Neither blocks nor log segments found. Creating new log.
I20260806 10:56:54.988610 14333 tablet_bootstrap.cc:492] T 1e65f12167a346a39018672ca5370d74 P a2291f6829ab46b084f98b6f7e974db4: Bootstrap starting.
I20260806 10:56:54.989840 14332 log.cc:824] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2: Log is configured to *not* fsync() on all Append() calls
I20260806 10:56:54.992138 14336 tablet_bootstrap.cc:492] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5: Bootstrap starting.
I20260806 10:56:54.992271 14333 tablet_bootstrap.cc:654] T 1e65f12167a346a39018672ca5370d74 P a2291f6829ab46b084f98b6f7e974db4: Neither blocks nor log segments found. Creating new log.
I20260806 10:56:54.993325 14332 tablet_bootstrap.cc:492] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2: No bootstrap required, opened a new log
I20260806 10:56:54.993342 14333 log.cc:824] T 1e65f12167a346a39018672ca5370d74 P a2291f6829ab46b084f98b6f7e974db4: Log is configured to *not* fsync() on all Append() calls
I20260806 10:56:54.993655 14332 ts_tablet_manager.cc:1397] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260806 10:56:54.995850 14336 tablet_bootstrap.cc:654] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5: Neither blocks nor log segments found. Creating new log.
I20260806 10:56:54.996411 14333 tablet_bootstrap.cc:492] T 1e65f12167a346a39018672ca5370d74 P a2291f6829ab46b084f98b6f7e974db4: No bootstrap required, opened a new log
I20260806 10:56:54.996691 14333 ts_tablet_manager.cc:1397] T 1e65f12167a346a39018672ca5370d74 P a2291f6829ab46b084f98b6f7e974db4: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260806 10:56:54.997025 14336 log.cc:824] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5: Log is configured to *not* fsync() on all Append() calls
I20260806 10:56:55.000187 14336 tablet_bootstrap.cc:492] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5: No bootstrap required, opened a new log
I20260806 10:56:55.000473 14336 ts_tablet_manager.cc:1397] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260806 10:56:55.003681 14332 raft_consensus.cc:348] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } } peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } }
I20260806 10:56:55.004186 14332 raft_consensus.cc:374] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:56:55.003980 14333 raft_consensus.cc:348] T 1e65f12167a346a39018672ca5370d74 P a2291f6829ab46b084f98b6f7e974db4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } } peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } }
I20260806 10:56:55.004283 14332 raft_consensus.cc:729] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d2c5277ea7546a3bbdef1d58c93fba2, State: Initialized, Role: FOLLOWER
I20260806 10:56:55.004375 14333 raft_consensus.cc:374] T 1e65f12167a346a39018672ca5370d74 P a2291f6829ab46b084f98b6f7e974db4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:56:55.004483 14333 raft_consensus.cc:729] T 1e65f12167a346a39018672ca5370d74 P a2291f6829ab46b084f98b6f7e974db4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2291f6829ab46b084f98b6f7e974db4, State: Initialized, Role: FOLLOWER
I20260806 10:56:55.005046 14333 consensus_queue.cc:260] T 1e65f12167a346a39018672ca5370d74 P a2291f6829ab46b084f98b6f7e974db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } } peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } }
I20260806 10:56:55.005059 14332 consensus_queue.cc:260] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } } peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } }
I20260806 10:56:55.006269 14176 heartbeater.cc:503] Master 127.30.218.62:42721 was elected leader, sending a full tablet report...
I20260806 10:56:55.006995 14333 ts_tablet_manager.cc:1428] T 1e65f12167a346a39018672ca5370d74 P a2291f6829ab46b084f98b6f7e974db4: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20260806 10:56:55.009109 14332 ts_tablet_manager.cc:1428] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
I20260806 10:56:55.010991 14336 raft_consensus.cc:348] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } } peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } }
I20260806 10:56:55.011549 14336 raft_consensus.cc:374] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:56:55.011657 14336 raft_consensus.cc:729] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae6994110a3b4d93bca7fadcaebafdc5, State: Initialized, Role: FOLLOWER
I20260806 10:56:55.012358 14336 consensus_queue.cc:260] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } } peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } }
I20260806 10:56:55.013587 14312 heartbeater.cc:503] Master 127.30.218.62:42721 was elected leader, sending a full tablet report...
I20260806 10:56:55.015246 14336 ts_tablet_manager.cc:1428] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
W20260806 10:56:55.131642 14039 tablet.cc:2369] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:56:55.134126 14313 tablet.cc:2369] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:56:55.134817 14177 tablet.cc:2369] T 1e65f12167a346a39018672ca5370d74 P a2291f6829ab46b084f98b6f7e974db4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:56:55.176923 14341 raft_consensus.cc:482] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:56:55.177250 14341 raft_consensus.cc:504] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } } peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } }
I20260806 10:56:55.179039 14341 leader_election.cc:288] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d2c5277ea7546a3bbdef1d58c93fba2 (127.30.218.1:43791), a2291f6829ab46b084f98b6f7e974db4 (127.30.218.2:33851)
I20260806 10:56:55.184906 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:55.179637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:43791 (local address 127.30.218.3:56471)
0806 10:56:55.179810 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:55.179826 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:55.179965 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:55.180343 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:55.180347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:55.180390 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:56:55.180561 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:55.180569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:55.181803 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:55.181808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:55.183120 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:55.183133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:55.184194 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:55.184200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:55.184383 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:55.184408 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 10:56:55.184490 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 10:56:55.184626 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 10:56:55.184911 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:55.179467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:56471 (local address 127.30.218.1:43791)
0806 10:56:55.179714 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:55.179720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:55.179912 (+ 192us) server_negotiation.cc:407] Connection header received
0806 10:56:55.180111 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:55.180117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:55.180199 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:55.180290 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:56:55.180718 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:55.181678 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:55.183282 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:55.184047 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:55.184194 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:55.184272 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:55.184537 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 10:56:55.184670 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 10:56:55.186036 14342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:55.179659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:33851 (local address 127.30.218.3:43971)
0806 10:56:55.180898 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:55.180913 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:55.181012 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:55.181307 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:55.181310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:55.181327 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:56:55.181663 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:55.181672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:55.182697 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:55.182701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:55.184051 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:55.184065 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:55.185299 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:55.185305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:55.185506 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:55.185540 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 10:56:55.185639 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 10:56:55.185737 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1143,"mutex_wait_us":407,"thread_start_us":497,"threads_started":1}
I20260806 10:56:55.186141 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:55.179654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:43971 (local address 127.30.218.2:33851)
0806 10:56:55.179864 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:55.179870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:55.181006 (+ 1136us) server_negotiation.cc:407] Connection header received
0806 10:56:55.181119 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:55.181125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:55.181205 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:55.181296 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:56:55.181831 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:55.182546 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:55.184220 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:55.185044 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:55.185214 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:55.185281 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:55.185711 (+ 430us) server_negotiation.cc:299] Negotiation successful
0806 10:56:55.185866 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 10:56:55.186219 13992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e65f12167a346a39018672ca5370d74" candidate_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" is_pre_election: true
I20260806 10:56:55.186911 13992 raft_consensus.cc:2454] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae6994110a3b4d93bca7fadcaebafdc5 in term 0.
I20260806 10:56:55.187084 14128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e65f12167a346a39018672ca5370d74" candidate_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2291f6829ab46b084f98b6f7e974db4" is_pre_election: true
I20260806 10:56:55.187587 14128 raft_consensus.cc:2454] T 1e65f12167a346a39018672ca5370d74 P a2291f6829ab46b084f98b6f7e974db4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae6994110a3b4d93bca7fadcaebafdc5 in term 0.
I20260806 10:56:55.187774 14199 leader_election.cc:302] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d2c5277ea7546a3bbdef1d58c93fba2, ae6994110a3b4d93bca7fadcaebafdc5; no voters:
I20260806 10:56:55.188192 14341 raft_consensus.cc:2781] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:56:55.188323 14341 raft_consensus.cc:482] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:56:55.188421 14341 raft_consensus.cc:3037] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:56:55.190691 14341 raft_consensus.cc:504] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } } peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } }
I20260806 10:56:55.191303 14341 leader_election.cc:288] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3d2c5277ea7546a3bbdef1d58c93fba2 (127.30.218.1:43791), a2291f6829ab46b084f98b6f7e974db4 (127.30.218.2:33851)
I20260806 10:56:55.191949 14128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e65f12167a346a39018672ca5370d74" candidate_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2291f6829ab46b084f98b6f7e974db4"
I20260806 10:56:55.191951 13992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e65f12167a346a39018672ca5370d74" candidate_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2"
I20260806 10:56:55.192204 14128 raft_consensus.cc:3037] T 1e65f12167a346a39018672ca5370d74 P a2291f6829ab46b084f98b6f7e974db4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:56:55.192205 13992 raft_consensus.cc:3037] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:56:55.195526 14128 raft_consensus.cc:2454] T 1e65f12167a346a39018672ca5370d74 P a2291f6829ab46b084f98b6f7e974db4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae6994110a3b4d93bca7fadcaebafdc5 in term 1.
I20260806 10:56:55.195526 13992 raft_consensus.cc:2454] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae6994110a3b4d93bca7fadcaebafdc5 in term 1.
I20260806 10:56:55.196017 14199 leader_election.cc:302] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d2c5277ea7546a3bbdef1d58c93fba2, ae6994110a3b4d93bca7fadcaebafdc5; no voters:
I20260806 10:56:55.196362 14341 raft_consensus.cc:2781] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:56:55.196812 14341 raft_consensus.cc:686] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5 [term 1 LEADER]: Becoming Leader. State: Replica: ae6994110a3b4d93bca7fadcaebafdc5, State: Running, Role: LEADER
I20260806 10:56:55.197302 14341 consensus_queue.cc:237] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } } peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } }
I20260806 10:56:55.202303 13848 catalog_manager.cc:5495] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae6994110a3b4d93bca7fadcaebafdc5 (127.30.218.3). New cstate: current_term: 1 leader_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } health_report { overall_health: UNKNOWN } } }
I20260806 10:56:55.265709 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:55.260031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60548 (local address 127.30.218.3:41919)
0806 10:56:55.260250 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:55.260254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:55.260275 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:56:55.260442 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:55.260446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:55.260533 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:55.260672 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:56:55.261001 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:55.261789 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:55.263282 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:55.263695 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:55.263791 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:55.264332 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:56:55.265014 (+ 682us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:56:55.265202 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:55.265302 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 10:56:55.265397 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 10:56:55.274497 14128 raft_consensus.cc:1261] T 1e65f12167a346a39018672ca5370d74 P a2291f6829ab46b084f98b6f7e974db4 [term 1 FOLLOWER]: Refusing update from remote peer ae6994110a3b4d93bca7fadcaebafdc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:56:55.274498 13992 raft_consensus.cc:1261] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Refusing update from remote peer ae6994110a3b4d93bca7fadcaebafdc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:56:55.275532 14343 consensus_queue.cc:1035] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:56:55.275876 14341 consensus_queue.cc:1035] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:56:55.286109 14348 mvcc.cc:205] Tried to move back new op lower bound from 7315512587353477120 to 7315512587051986944. Current Snapshot: MvccSnapshot[applied={T|T < 7315512587353477120}]
I20260806 10:56:55.286370 14346 mvcc.cc:205] Tried to move back new op lower bound from 7315512587353477120 to 7315512587051986944. Current Snapshot: MvccSnapshot[applied={T|T < 7315512587353477120}]
I20260806 10:56:55.301833 13848 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:56:55.302111 13848 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:56:55.306106 13848 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67: Sending DeleteTablet for 3 replicas of tablet 1e65f12167a346a39018672ca5370d74
I20260806 10:56:55.307479 14244 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e65f12167a346a39018672ca5370d74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:56:55 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:60532
I20260806 10:56:55.307428 13970 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e65f12167a346a39018672ca5370d74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:56:55 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:59292
I20260806 10:56:55.308038 14107 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e65f12167a346a39018672ca5370d74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:56:55 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:46210
I20260806 10:56:55.308861 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 13886
I20260806 10:56:55.323822 14356 tablet_replica.cc:330] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5: stopping tablet replica
I20260806 10:56:55.324954 14356 raft_consensus.cc:2229] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:56:55.325981 14356 raft_consensus.cc:2258] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:56:55.327133 14355 tablet_replica.cc:330] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2: stopping tablet replica
I20260806 10:56:55.329056 14356 ts_tablet_manager.cc:1905] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:56:55.329273 14355 raft_consensus.cc:2229] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:56:55.331441 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:55.309893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59302 (local address 127.30.218.1:43791)
0806 10:56:55.310555 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:55.310560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:55.310580 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:56:55.310643 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:55.310649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:55.310741 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:55.310837 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:55.313063 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:55.314405 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:55.316661 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:55.317424 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:55.317598 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:55.321911 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:55.321937 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:55.321960 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:55.322017 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:55.327104 (+ 5087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:55.328259 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:55.328271 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:55.328275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:55.328501 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:55.328964 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:55.328970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:55.328973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:55.330071 (+ 1098us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:55.330407 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:55.330726 (+ 319us) server_negotiation.cc:299] Negotiation successful
0806 10:56:55.331008 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524}
I20260806 10:56:55.332145 14355 raft_consensus.cc:2258] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:56:55.332417 14033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:56:55.335857 14356 ts_tablet_manager.cc:1918] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:56:55.336041 14356 log.cc:1196] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/1e65f12167a346a39018672ca5370d74
I20260806 10:56:55.336812 14356 ts_tablet_manager.cc:1939] T 1e65f12167a346a39018672ca5370d74 P ae6994110a3b4d93bca7fadcaebafdc5: Deleting consensus metadata
I20260806 10:56:55.338229 14357 tablet_replica.cc:330] T 1e65f12167a346a39018672ca5370d74 P a2291f6829ab46b084f98b6f7e974db4: stopping tablet replica
I20260806 10:56:55.339277 13833 catalog_manager.cc:4841] TS ae6994110a3b4d93bca7fadcaebafdc5 (127.30.218.3:41919): tablet 1e65f12167a346a39018672ca5370d74 (table test-table [id=ec940585565a4d28bd545ec703448e70]) successfully deleted
I20260806 10:56:55.338760 14357 raft_consensus.cc:2229] T 1e65f12167a346a39018672ca5370d74 P a2291f6829ab46b084f98b6f7e974db4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:56:55.340328 14357 raft_consensus.cc:2258] T 1e65f12167a346a39018672ca5370d74 P a2291f6829ab46b084f98b6f7e974db4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:56:55.344580 14357 ts_tablet_manager.cc:1905] T 1e65f12167a346a39018672ca5370d74 P a2291f6829ab46b084f98b6f7e974db4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:56:55.352679 14357 ts_tablet_manager.cc:1918] T 1e65f12167a346a39018672ca5370d74 P a2291f6829ab46b084f98b6f7e974db4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:56:55.352919 14357 log.cc:1196] T 1e65f12167a346a39018672ca5370d74 P a2291f6829ab46b084f98b6f7e974db4: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/1e65f12167a346a39018672ca5370d74
I20260806 10:56:55.353593 14357 ts_tablet_manager.cc:1939] T 1e65f12167a346a39018672ca5370d74 P a2291f6829ab46b084f98b6f7e974db4: Deleting consensus metadata
I20260806 10:56:55.355837 13830 catalog_manager.cc:4841] TS a2291f6829ab46b084f98b6f7e974db4 (127.30.218.2:33851): tablet 1e65f12167a346a39018672ca5370d74 (table test-table [id=ec940585565a4d28bd545ec703448e70]) successfully deleted
I20260806 10:56:55.451920 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 13886
I20260806 10:56:55.469907 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 14042
W20260806 10:56:55.469836 13834 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.218.1:43791 (error 108)
I20260806 10:56:55.471920 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:55.471304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:43791 (local address 127.0.0.1:59310)
0806 10:56:55.471579 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:55.471732 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260806 10:56:55.472196 13834 catalog_manager.cc:4584] TS 3d2c5277ea7546a3bbdef1d58c93fba2 (127.30.218.1:43791): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e65f12167a346a39018672ca5370d74: Network error: Client connection negotiation failed: client connection to 127.30.218.1:43791: connect: Connection refused (error 111)
I20260806 10:56:55.488224 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:55.471176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46214 (local address 127.30.218.2:33851)
0806 10:56:55.471370 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:55.471376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:55.471397 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:56:55.471468 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:55.471473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:55.471557 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:55.471662 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:55.473868 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:55.474665 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:55.476774 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:55.477213 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:55.477324 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:55.481155 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:55.481177 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:55.481192 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:55.481238 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:55.484546 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:55.485684 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:55.485694 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:55.485696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:55.485853 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:55.486327 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:55.486331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:55.486333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:55.486785 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:55.487033 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:55.487513 (+ 480us) server_negotiation.cc:299] Negotiation successful
0806 10:56:55.487743 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 10:56:55.489005 14170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:56:55.526253 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:55.525607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:43791 (local address 127.0.0.1:59312)
0806 10:56:55.525908 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:55.526053 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260806 10:56:55.527581 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:55.527104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:43791 (local address 127.0.0.1:59318)
0806 10:56:55.527298 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:55.527407 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 10:56:55.582072 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 14042
I20260806 10:56:55.588405 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:55.587740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:43791 (local address 127.0.0.1:59334)
0806 10:56:55.588014 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:55.588152 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260806 10:56:55.589738 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:55.589281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:43791 (local address 127.0.0.1:59336)
0806 10:56:55.589473 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:55.589580 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 10:56:55.600500 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 14179
I20260806 10:56:55.615967 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:55.601322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.30.218.3:41919)
0806 10:56:55.601582 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:55.601588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:55.601613 (+ 25us) server_negotiation.cc:407] Connection header received
0806 10:56:55.601726 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:55.601732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:55.601824 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:55.601910 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:55.603354 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:55.604211 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:55.605648 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:55.606167 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:55.606292 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:55.609078 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:55.609098 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:55.609116 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:55.609162 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:55.612410 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:55.613280 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:55.613289 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:55.613291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:55.613556 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:55.614115 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:55.614119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:55.614121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:55.614563 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:55.614902 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:55.615294 (+ 392us) server_negotiation.cc:299] Negotiation successful
0806 10:56:55.615516 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 10:56:55.616797 14306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:56:55.670567 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:55.669985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:43791 (local address 127.0.0.1:59344)
0806 10:56:55.670243 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:55.670343 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260806 10:56:55.671901 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:55.671272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:43791 (local address 127.0.0.1:59354)
0806 10:56:55.671579 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:55.671735 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260806 10:56:55.745323 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 14179
I20260806 10:56:55.765240 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 13813
I20260806 10:56:55.766527 13879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:56:55.886093 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 13813
I20260806 10:56:55.909641 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:42721
--webserver_interface=127.30.218.62
--webserver_port=41045
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:43985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:42721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:56:56.274425 14363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:56:56.274812 14363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:56:56.274958 14363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:56:56.282868 14363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:56:56.283037 14363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:56:56.283131 14363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:56:56.283213 14363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:56:56.283275 14363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:56:56.294659 14363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:43985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:42721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:42721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=41045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:56:56.296201 14363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:56:56.297688 14363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:56:56.311113 14368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:56.311170 14371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:56.311154 14369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:56.311911 14363 server_base.cc:1034] running on GCE node
I20260806 10:56:56.312870 14363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:56:56.314621 14363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:56:56.315852 14363 hybrid_clock.cc:648] HybridClock initialized: now 1786013816315785 us; error 60 us; skew 500 ppm
I20260806 10:56:56.316253 14363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:56:56 dist-test-slave-49h3 krb5kdc[13254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013816, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:56.319684 14363 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:56:56.321707 14363 webserver.cc:460] Webserver started at http://127.30.218.62:41045/ using document root <none> and password file <none>
I20260806 10:56:56.322477 14363 fs_manager.cc:359] Metadata directory not provided
I20260806 10:56:56.322621 14363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:56:56.329093 14363 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260806 10:56:56.332537 14378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:56.334120 14363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 10:56:56.334333 14363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "1a866dfcd2fd4db3a6082469f1e47e67"
format_stamp: "Formatted at 2026-08-06 10:56:50 on dist-test-slave-49h3"
server_key: "57b918a3d7eeb3c385415ab2b4b7269e"
server_key_iv: "6d4b4e9646b87a0dd2e935c73a20c37e"
server_key_version: "encryptionkey@0"
I20260806 10:56:56.335414 14363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:56:56.372110 14363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:56:56.376900 14363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:56:56.377210 14363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:56:56.396323 14363 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:42721
I20260806 10:56:56.396344 14432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:42721 every 8 connection(s)
I20260806 10:56:56.398010 14363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:56:56.403476 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 14363
I20260806 10:56:56.404597 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:43791
--local_ip_for_outbound_sockets=127.30.218.1
--tserver_master_addrs=127.30.218.62:42721
--webserver_port=39225
--webserver_interface=127.30.218.1
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:43985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:56:56.406407 14433 sys_catalog.cc:263] Verifying existing consensus state
I20260806 10:56:56.411195 14433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67: Bootstrap starting.
I20260806 10:56:56.434508 14433 log.cc:824] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67: Log is configured to *not* fsync() on all Append() calls
I20260806 10:56:56.454041 14433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 10:56:56.454622 14433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67: Bootstrap complete.
I20260806 10:56:56.464066 14433 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a866dfcd2fd4db3a6082469f1e47e67" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42721 } }
I20260806 10:56:56.465057 14433 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a866dfcd2fd4db3a6082469f1e47e67, State: Initialized, Role: FOLLOWER
I20260806 10:56:56.465830 14433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a866dfcd2fd4db3a6082469f1e47e67" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42721 } }
I20260806 10:56:56.466104 14433 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:56:56.466233 14433 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:56:56.466408 14433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 2 FOLLOWER]: Advancing to term 3
I20260806 10:56:56.470626 14433 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a866dfcd2fd4db3a6082469f1e47e67" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42721 } }
I20260806 10:56:56.471230 14433 leader_election.cc:302] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a866dfcd2fd4db3a6082469f1e47e67; no voters:
I20260806 10:56:56.471885 14433 leader_election.cc:288] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 10:56:56.472038 14437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 10:56:56.473228 14437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [term 3 LEADER]: Becoming Leader. State: Replica: 1a866dfcd2fd4db3a6082469f1e47e67, State: Running, Role: LEADER
I20260806 10:56:56.473950 14437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a866dfcd2fd4db3a6082469f1e47e67" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42721 } }
I20260806 10:56:56.474349 14433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:56:56.476593 14439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a866dfcd2fd4db3a6082469f1e47e67. Latest consensus state: current_term: 3 leader_uuid: "1a866dfcd2fd4db3a6082469f1e47e67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a866dfcd2fd4db3a6082469f1e47e67" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42721 } } }
I20260806 10:56:56.477427 14439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [sys.catalog]: This master's current role is: LEADER
I20260806 10:56:56.477110 14438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1a866dfcd2fd4db3a6082469f1e47e67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a866dfcd2fd4db3a6082469f1e47e67" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42721 } } }
I20260806 10:56:56.478040 14438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a866dfcd2fd4db3a6082469f1e47e67 [sys.catalog]: This master's current role is: LEADER
I20260806 10:56:56.486167 14443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:56:56.496973 14443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:56:56.498965 14443 catalog_manager.cc:1259] Loaded cluster ID: 403cc141659e4e7bb759a4497f8007db
I20260806 10:56:56.499184 14443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 10:56:56.502050 14443 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff8a83bd9ad google::LogMessage::Fail() at ??:0
@ 0x7ff8a83c088a google::LogMessage::SendToLog() at ??:0
@ 0x7ff8a83bd4de google::LogMessage::Flush() at ??:0
@ 0x7ff8a83bea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff8bddf0512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff8bdded976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff8bde75610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff8bde75439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff8ab009345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff8ab00c91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff8ab00c799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff8aafe7eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff8ae2c16db start_thread at ??:0
@ 0x7ff8a725761f clone at ??:0
W20260806 10:56:56.812320 14435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:56:56.812696 14435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:56:56.812840 14435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:56:56.821003 14435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:56:56.821159 14435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:56:56.821331 14435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:56:56.832121 14435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:43985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:43791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=39225
--tserver_master_addrs=127.30.218.62:42721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:56:56.833679 14435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:56:56.835130 14435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:56:56.849328 14475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:56.849364 14479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:56.849291 14476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:56.850453 14435 server_base.cc:1034] running on GCE node
I20260806 10:56:56.851133 14435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:56:56.852252 14435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:56:56.853508 14435 hybrid_clock.cc:648] HybridClock initialized: now 1786013816853406 us; error 65 us; skew 500 ppm
I20260806 10:56:56.853858 14435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:56:56 dist-test-slave-49h3 krb5kdc[13254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013816, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:56.857218 14435 init.cc:377] Logged in from keytab as kudu/127.30.218.1@KRBTEST.COM (short username kudu)
I20260806 10:56:56.858989 14435 webserver.cc:460] Webserver started at http://127.30.218.1:39225/ using document root <none> and password file <none>
I20260806 10:56:56.859825 14435 fs_manager.cc:359] Metadata directory not provided
I20260806 10:56:56.859965 14435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:56:56.866128 14435 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 10:56:56.869350 14486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:56.870926 14435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:56:56.871116 14435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "3d2c5277ea7546a3bbdef1d58c93fba2"
format_stamp: "Formatted at 2026-08-06 10:56:50 on dist-test-slave-49h3"
server_key: "7d11dbb0738e614a2868f360eafdc805"
server_key_iv: "f6d5a7dd3dc78cf870bf802703579218"
server_key_version: "encryptionkey@0"
I20260806 10:56:56.872210 14435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:56:56.893276 14435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:56:56.897796 14435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:56:56.898092 14435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:56:56.899243 14435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:56:56.901376 14493 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260806 10:56:56.907769 14435 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 10:56:56.907953 14435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260806 10:56:56.908133 14435 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 10:56:56.911742 14435 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 10:56:56.911850 14435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20260806 10:56:56.912271 14493 tablet_bootstrap.cc:492] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2: Bootstrap starting.
I20260806 10:56:56.921996 14493 log.cc:824] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2: Log is configured to *not* fsync() on all Append() calls
I20260806 10:56:56.942340 14493 tablet_bootstrap.cc:492] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 10:56:56.943543 14493 tablet_bootstrap.cc:492] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2: Bootstrap complete.
I20260806 10:56:56.944227 14493 ts_tablet_manager.cc:1397] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.004s
I20260806 10:56:56.953040 14435 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:43791
I20260806 10:56:56.953051 14604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:43791 every 8 connection(s)
I20260806 10:56:56.955022 14435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:56:56.956452 14493 raft_consensus.cc:348] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } } peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } }
I20260806 10:56:56.957747 14493 raft_consensus.cc:729] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d2c5277ea7546a3bbdef1d58c93fba2, State: Initialized, Role: FOLLOWER
I20260806 10:56:56.958529 14493 consensus_queue.cc:260] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } } peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } }
I20260806 10:56:56.960364 14493 ts_tablet_manager.cc:1428] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260806 10:56:56.960909 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:56.958467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.1:34191)
0806 10:56:56.959553 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:56.959844 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":447,"threads_started":1}
W20260806 10:56:56.961640 14605 heartbeater.cc:644] Failed to heartbeat to 127.30.218.62:42721 (0 consecutive failures): Network error: Failed to ping master at 127.30.218.62:42721: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
I20260806 10:56:56.961959 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 14435
I20260806 10:56:56.962704 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:56.962283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.1:56077)
0806 10:56:56.962460 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:56.962557 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 10:56:56.962739 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:33851
--local_ip_for_outbound_sockets=127.30.218.2
--tserver_master_addrs=127.30.218.62:42721
--webserver_port=35963
--webserver_interface=127.30.218.2
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:43985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:56:56.963697 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:56.963304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.1:41257)
0806 10:56:56.963445 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:56.963534 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 10:56:56.963948 14605 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 10:56:57.314903 14609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:56:57.315244 14609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:56:57.315346 14609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:56:57.322924 14609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:56:57.323065 14609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:56:57.323185 14609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:56:57.333588 14609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:43985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:33851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=35963
--tserver_master_addrs=127.30.218.62:42721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:56:57.335032 14609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:56:57.336373 14609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:56:57.350297 14617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:57.350515 14615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:57.350394 14614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:57.351662 14609 server_base.cc:1034] running on GCE node
I20260806 10:56:57.352368 14609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:56:57.353545 14609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:56:57.354781 14609 hybrid_clock.cc:648] HybridClock initialized: now 1786013817354729 us; error 44 us; skew 500 ppm
I20260806 10:56:57.355192 14609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:56:57 dist-test-slave-49h3 krb5kdc[13254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013817, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:57.358727 14609 init.cc:377] Logged in from keytab as kudu/127.30.218.2@KRBTEST.COM (short username kudu)
I20260806 10:56:57.360334 14609 webserver.cc:460] Webserver started at http://127.30.218.2:35963/ using document root <none> and password file <none>
I20260806 10:56:57.361064 14609 fs_manager.cc:359] Metadata directory not provided
I20260806 10:56:57.361177 14609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:56:57.367221 14609 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 10:56:57.370246 14625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:57.371794 14609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:56:57.371980 14609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "a2291f6829ab46b084f98b6f7e974db4"
format_stamp: "Formatted at 2026-08-06 10:56:51 on dist-test-slave-49h3"
server_key: "3b1505198463f668cac7d66d6d2bebaf"
server_key_iv: "456df9b2734955974a8444d92fa3c579"
server_key_version: "encryptionkey@0"
I20260806 10:56:57.373083 14609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:56:57.400748 14609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:56:57.405481 14609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:56:57.405784 14609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:56:57.406899 14609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:56:57.408365 14609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:56:57.408470 14609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:57.408591 14609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:56:57.408663 14609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:57.440764 14609 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:33851
I20260806 10:56:57.440824 14740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:33851 every 8 connection(s)
I20260806 10:56:57.442370 14609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:56:57.446161 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 14609
I20260806 10:56:57.446908 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:41919
--local_ip_for_outbound_sockets=127.30.218.3
--tserver_master_addrs=127.30.218.62:42721
--webserver_port=43225
--webserver_interface=127.30.218.3
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:43985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:56:57.447494 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:57.445440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.2:37379)
0806 10:56:57.446307 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:57.446559 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":312,"threads_started":1}
W20260806 10:56:57.448254 14741 heartbeater.cc:644] Failed to heartbeat to 127.30.218.62:42721 (0 consecutive failures): Network error: Failed to ping master at 127.30.218.62:42721: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
I20260806 10:56:57.449136 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:57.448777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.2:38319)
0806 10:56:57.448937 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:57.449013 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 10:56:57.450075 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:57.449735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.2:34545)
0806 10:56:57.449861 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:57.449940 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 10:56:57.450321 14741 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 10:56:57.801020 14744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:56:57.801371 14744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:56:57.801505 14744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:56:57.809239 14744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:56:57.809378 14744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:56:57.809541 14744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:56:57.819984 14744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:43985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:41919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=43225
--tserver_master_addrs=127.30.218.62:42721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:56:57.821350 14744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:56:57.822767 14744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:56:57.836889 14749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:57.836858 14752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:56:57.836918 14750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:56:57.837617 14744 server_base.cc:1034] running on GCE node
I20260806 10:56:57.838347 14744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:56:57.839417 14744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:56:57.840646 14744 hybrid_clock.cc:648] HybridClock initialized: now 1786013817840592 us; error 49 us; skew 500 ppm
I20260806 10:56:57.840981 14744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:56:57 dist-test-slave-49h3 krb5kdc[13254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013817, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:56:57.844340 14744 init.cc:377] Logged in from keytab as kudu/127.30.218.3@KRBTEST.COM (short username kudu)
I20260806 10:56:57.846148 14744 webserver.cc:460] Webserver started at http://127.30.218.3:43225/ using document root <none> and password file <none>
I20260806 10:56:57.846966 14744 fs_manager.cc:359] Metadata directory not provided
I20260806 10:56:57.847173 14744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:56:57.853310 14744 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 10:56:57.856417 14760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:57.858202 14744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:56:57.858417 14744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "ae6994110a3b4d93bca7fadcaebafdc5"
format_stamp: "Formatted at 2026-08-06 10:56:51 on dist-test-slave-49h3"
server_key: "25ee91cc2976bb4fb22f1f6fb0dc427e"
server_key_iv: "5fb92550d947966e28c49e76a3ac3f10"
server_key_version: "encryptionkey@0"
I20260806 10:56:57.859601 14744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:56:57.888691 14744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:56:57.893853 14744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:56:57.894157 14744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:56:57.895311 14744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:56:57.896853 14744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:56:57.896950 14744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:57.897065 14744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:56:57.897130 14744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:56:57.930684 14744 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:41919
I20260806 10:56:57.930749 14873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:41919 every 8 connection(s)
I20260806 10:56:57.932324 14744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:56:57.939973 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:57.935724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.3:53731)
0806 10:56:57.938220 (+ 2496us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:57.938600 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2095,"thread_start_us":1855,"threads_started":1}
I20260806 10:56:57.940165 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 14744
W20260806 10:56:57.941920 14874 heartbeater.cc:644] Failed to heartbeat to 127.30.218.62:42721 (0 consecutive failures): Network error: Failed to ping master at 127.30.218.62:42721: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
I20260806 10:56:57.943238 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:57.942662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.3:43025)
0806 10:56:57.942943 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:57.943059 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260806 10:56:57.944751 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:57.944273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.3:40117)
0806 10:56:57.944453 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:57.944575 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 10:56:57.945072 14874 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 10:56:57.965690 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:57.964733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.1:44529)
0806 10:56:57.965263 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:57.965407 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":244,"threads_started":1}
I20260806 10:56:58.333851 14879 raft_consensus.cc:482] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:56:58.334172 14879 raft_consensus.cc:504] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } } peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } }
I20260806 10:56:58.335876 14879 leader_election.cc:288] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae6994110a3b4d93bca7fadcaebafdc5 (127.30.218.3:41919), a2291f6829ab46b084f98b6f7e974db4 (127.30.218.2:33851)
Aug 06 10:56:58 dist-test-slave-49h3 krb5kdc[13254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013816, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for kudu/127.30.218.3@KRBTEST.COM
Aug 06 10:56:58 dist-test-slave-49h3 krb5kdc[13254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013816, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for kudu/127.30.218.2@KRBTEST.COM
I20260806 10:56:58.363981 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:58.336374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:33851 (local address 127.30.218.1:48349)
0806 10:56:58.338675 (+ 2301us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:58.338692 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:58.338829 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:58.341413 (+ 2584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:58.341425 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:58.342296 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:56:58.343038 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:58.343066 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:58.344663 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:58.344667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:58.346187 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:58.346203 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:58.347262 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:58.347270 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:58.347485 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:58.353658 (+ 6173us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:56:58.353695 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:56:58.356695 (+ 3000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:56:58.360132 (+ 3437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:58.360137 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:58.360149 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:58.360752 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:58.361223 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:58.361228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:58.361232 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:58.361437 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:58.362289 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:56:58.362297 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:56:58.362613 (+ 316us) client_negotiation.cc:769] Sending connection context
0806 10:56:58.362871 (+ 258us) client_negotiation.cc:240] Negotiation successful
0806 10:56:58.363383 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2171,"mutex_wait_us":444,"spinlock_wait_cycles":34432,"thread_start_us":1718,"threads_started":1}
I20260806 10:56:58.364058 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:58.336356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:41919 (local address 127.30.218.1:39957)
0806 10:56:58.336645 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:58.336687 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 10:56:58.337342 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:56:58.341414 (+ 4072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:56:58.341425 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:56:58.342273 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:56:58.343038 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:58.343066 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:58.344654 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:58.344658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:58.346168 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:56:58.346183 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:56:58.347214 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:56:58.347221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:56:58.347484 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:58.353658 (+ 6174us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:56:58.353692 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:56:58.356390 (+ 2698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:56:58.360120 (+ 3730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:58.360131 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:58.360147 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:58.360752 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:58.361215 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:56:58.361220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:56:58.361224 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:56:58.361420 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:56:58.362291 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:56:58.362298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:56:58.362613 (+ 315us) client_negotiation.cc:769] Sending connection context
0806 10:56:58.362870 (+ 257us) client_negotiation.cc:240] Negotiation successful
0806 10:56:58.363646 (+ 776us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":375,"spinlock_wait_cycles":64128}
I20260806 10:56:58.364708 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:58.336493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:39957 (local address 127.30.218.3:41919)
0806 10:56:58.338638 (+ 2145us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:58.338651 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:58.338701 (+ 50us) server_negotiation.cc:407] Connection header received
0806 10:56:58.340403 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:58.340416 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:58.340838 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:58.341229 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:58.343260 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:58.344462 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:58.346317 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:58.346980 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:58.347170 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:58.356829 (+ 9659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:58.356844 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:58.356862 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:58.356918 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:58.359937 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:58.360911 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:58.360919 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:58.360924 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:58.361100 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:58.361574 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:58.361585 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:58.361587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:58.362151 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:58.362326 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:58.363149 (+ 823us) server_negotiation.cc:299] Negotiation successful
0806 10:56:58.364291 (+ 1142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1956,"thread_start_us":1775,"threads_started":1}
I20260806 10:56:58.365101 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:58.336555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:48349 (local address 127.30.218.2:33851)
0806 10:56:58.338183 (+ 1628us) server_negotiation.cc:206] Beginning negotiation
0806 10:56:58.338200 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 10:56:58.338803 (+ 603us) server_negotiation.cc:407] Connection header received
0806 10:56:58.340353 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:56:58.340362 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:56:58.340899 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:56:58.341238 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:56:58.343261 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:58.344448 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:58.346347 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:56:58.347001 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:56:58.347180 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:56:58.357025 (+ 9845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:56:58.357034 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:56:58.357047 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:56:58.357092 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:56:58.359975 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:58.360911 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:58.360919 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:58.360924 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:58.361105 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:56:58.361581 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:56:58.361586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:56:58.361589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:56:58.362148 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:56:58.362333 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 10:56:58.363622 (+ 1289us) server_negotiation.cc:299] Negotiation successful
0806 10:56:58.364734 (+ 1112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1436,"thread_start_us":1322,"threads_started":1}
W20260806 10:56:58.367705 14487 leader_election.cc:341] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2291f6829ab46b084f98b6f7e974db4 (127.30.218.2:33851): Not found: Tablet not found: 1e65f12167a346a39018672ca5370d74
W20260806 10:56:58.367928 14489 leader_election.cc:341] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae6994110a3b4d93bca7fadcaebafdc5 (127.30.218.3:41919): Not found: Tablet not found: 1e65f12167a346a39018672ca5370d74
I20260806 10:56:58.368076 14489 leader_election.cc:302] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d2c5277ea7546a3bbdef1d58c93fba2; no voters: a2291f6829ab46b084f98b6f7e974db4, ae6994110a3b4d93bca7fadcaebafdc5
I20260806 10:56:58.368470 14879 raft_consensus.cc:2726] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 10:56:58.451894 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:58.450944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.2:46807)
0806 10:56:58.451445 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:58.451653 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":228,"threads_started":1}
I20260806 10:56:58.946718 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:58.945757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.3:58061)
0806 10:56:58.946326 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:58.946482 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":273,"threads_started":1}
I20260806 10:56:58.967621 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:58.966739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.1:37079)
0806 10:56:58.967276 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:58.967406 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":289,"threads_started":1}
I20260806 10:56:59.453737 14887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:59.452884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.2:49807)
0806 10:56:59.453403 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:59.453533 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":251,"threads_started":1}
I20260806 10:56:59.948650 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:59.947818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.3:40975)
0806 10:56:59.948333 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:59.948440 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":263,"threads_started":1}
I20260806 10:56:59.969900 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:56:59.968818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.1:35879)
0806 10:56:59.969468 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0806 10:56:59.969619 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":314,"threads_started":1}
I20260806 10:57:00.455602 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:00.454773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.2:57577)
0806 10:57:00.455293 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:00.455398 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":254,"threads_started":1}
I20260806 10:57:00.594125 14892 raft_consensus.cc:482] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:57:00.594388 14892 raft_consensus.cc:504] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } } peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } }
I20260806 10:57:00.595105 14892 leader_election.cc:288] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae6994110a3b4d93bca7fadcaebafdc5 (127.30.218.3:41919), a2291f6829ab46b084f98b6f7e974db4 (127.30.218.2:33851)
W20260806 10:57:00.596153 14487 leader_election.cc:341] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2291f6829ab46b084f98b6f7e974db4 (127.30.218.2:33851): Not found: Tablet not found: 1e65f12167a346a39018672ca5370d74
W20260806 10:57:00.596362 14489 leader_election.cc:341] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae6994110a3b4d93bca7fadcaebafdc5 (127.30.218.3:41919): Not found: Tablet not found: 1e65f12167a346a39018672ca5370d74
I20260806 10:57:00.596453 14489 leader_election.cc:302] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d2c5277ea7546a3bbdef1d58c93fba2; no voters: a2291f6829ab46b084f98b6f7e974db4, ae6994110a3b4d93bca7fadcaebafdc5
I20260806 10:57:00.596710 14892 raft_consensus.cc:2726] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 10:57:00.950383 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:00.949674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.3:51385)
0806 10:57:00.950124 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:00.950218 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":227,"threads_started":1}
I20260806 10:57:00.971781 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:00.970947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.1:49601)
0806 10:57:00.971470 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:00.971567 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":240,"threads_started":1}
I20260806 10:57:01.457559 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:01.456665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.2:38117)
0806 10:57:01.457198 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:01.457299 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":259,"threads_started":1}
I20260806 10:57:01.952316 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:01.951370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.3:42375)
0806 10:57:01.951972 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:01.952080 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":295,"threads_started":1}
I20260806 10:57:01.974036 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:01.972967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.1:57843)
0806 10:57:01.973673 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:01.973801 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":250,"threads_started":1}
I20260806 10:57:02.459481 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:02.458615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.2:56547)
0806 10:57:02.459143 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:02.459252 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":277,"threads_started":1}
I20260806 10:57:02.954598 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:02.953531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.3:53927)
0806 10:57:02.954217 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:02.954361 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":303,"threads_started":1}
I20260806 10:57:02.976040 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:02.975178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.1:37789)
0806 10:57:02.975727 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:02.975840 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":275,"threads_started":1}
I20260806 10:57:03.129529 14902 raft_consensus.cc:482] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:57:03.129762 14902 raft_consensus.cc:504] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } } peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } }
I20260806 10:57:03.130487 14902 leader_election.cc:288] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae6994110a3b4d93bca7fadcaebafdc5 (127.30.218.3:41919), a2291f6829ab46b084f98b6f7e974db4 (127.30.218.2:33851)
W20260806 10:57:03.131702 14489 leader_election.cc:341] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae6994110a3b4d93bca7fadcaebafdc5 (127.30.218.3:41919): Not found: Tablet not found: 1e65f12167a346a39018672ca5370d74
W20260806 10:57:03.131915 14487 leader_election.cc:341] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2291f6829ab46b084f98b6f7e974db4 (127.30.218.2:33851): Not found: Tablet not found: 1e65f12167a346a39018672ca5370d74
I20260806 10:57:03.132139 14487 leader_election.cc:302] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d2c5277ea7546a3bbdef1d58c93fba2; no voters: a2291f6829ab46b084f98b6f7e974db4, ae6994110a3b4d93bca7fadcaebafdc5
I20260806 10:57:03.132472 14902 raft_consensus.cc:2726] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 10:57:03.461410 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:03.460565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.2:35899)
0806 10:57:03.461091 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:03.461208 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":253,"threads_started":1}
I20260806 10:57:03.958941 14904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:03.955735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.3:50407)
0806 10:57:03.958532 (+ 2797us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:03.958673 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2647,"thread_start_us":2487,"threads_started":1}
I20260806 10:57:03.978122 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:03.977199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.1:49521)
0806 10:57:03.977716 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:03.977875 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"spinlock_wait_cycles":4864,"thread_start_us":381,"threads_started":1}
I20260806 10:57:04.463948 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:04.462954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.2:41563)
0806 10:57:04.463595 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:04.463697 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":235,"threads_started":1}
I20260806 10:57:04.682376 14907 raft_consensus.cc:482] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:57:04.682590 14907 raft_consensus.cc:504] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } } peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } }
I20260806 10:57:04.683331 14907 leader_election.cc:288] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae6994110a3b4d93bca7fadcaebafdc5 (127.30.218.3:41919), a2291f6829ab46b084f98b6f7e974db4 (127.30.218.2:33851)
W20260806 10:57:04.684433 14487 leader_election.cc:341] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2291f6829ab46b084f98b6f7e974db4 (127.30.218.2:33851): Not found: Tablet not found: 1e65f12167a346a39018672ca5370d74
W20260806 10:57:04.684623 14489 leader_election.cc:341] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae6994110a3b4d93bca7fadcaebafdc5 (127.30.218.3:41919): Not found: Tablet not found: 1e65f12167a346a39018672ca5370d74
I20260806 10:57:04.684788 14489 leader_election.cc:302] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d2c5277ea7546a3bbdef1d58c93fba2; no voters: a2291f6829ab46b084f98b6f7e974db4, ae6994110a3b4d93bca7fadcaebafdc5
I20260806 10:57:04.685065 14907 raft_consensus.cc:2726] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 10:57:04.960997 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:04.960074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.3:42649)
0806 10:57:04.960667 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:04.960774 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":263,"threads_started":1}
I20260806 10:57:04.979916 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:04.979070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.1:44057)
0806 10:57:04.979605 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:04.979710 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":257,"threads_started":1}
I20260806 10:57:05.465929 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:05.465125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.2:55279)
0806 10:57:05.465637 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:05.465729 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":253,"threads_started":1}
I20260806 10:57:05.963069 14911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:05.962122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.3:53861)
0806 10:57:05.962724 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:05.962829 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":330,"threads_started":1}
I20260806 10:57:05.981885 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:05.980889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.1:35821)
0806 10:57:05.981526 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:05.981630 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":300,"threads_started":1}
I20260806 10:57:06.467871 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:06.466990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.2:59017)
0806 10:57:06.467554 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:06.467662 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":261,"threads_started":1}
I20260806 10:57:06.965142 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:06.964129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.3:34321)
0806 10:57:06.964814 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:06.964935 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":366,"threads_started":1}
I20260806 10:57:06.984043 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:06.983129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.1:56749)
0806 10:57:06.983699 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:06.983804 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":308,"threads_started":1}
I20260806 10:57:07.470059 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:07.469065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.2:37735)
0806 10:57:07.469682 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:07.469799 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":348,"threads_started":1}
I20260806 10:57:07.967079 14917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:07.966246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.3:50791)
0806 10:57:07.966759 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:07.966865 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":249,"threads_started":1}
I20260806 10:57:07.986235 14918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:07.985206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.1:42943)
0806 10:57:07.985860 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:07.985979 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":372,"threads_started":1}
I20260806 10:57:08.472111 14919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:08.471157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.2:44365)
0806 10:57:08.471714 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:08.471893 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":242,"threads_started":1}
I20260806 10:57:08.797262 14920 raft_consensus.cc:482] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:57:08.797500 14920 raft_consensus.cc:504] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d2c5277ea7546a3bbdef1d58c93fba2" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43791 } } peers { permanent_uuid: "ae6994110a3b4d93bca7fadcaebafdc5" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41919 } } peers { permanent_uuid: "a2291f6829ab46b084f98b6f7e974db4" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33851 } }
I20260806 10:57:08.798220 14920 leader_election.cc:288] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae6994110a3b4d93bca7fadcaebafdc5 (127.30.218.3:41919), a2291f6829ab46b084f98b6f7e974db4 (127.30.218.2:33851)
W20260806 10:57:08.799173 14487 leader_election.cc:341] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2291f6829ab46b084f98b6f7e974db4 (127.30.218.2:33851): Not found: Tablet not found: 1e65f12167a346a39018672ca5370d74
W20260806 10:57:08.799341 14489 leader_election.cc:341] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae6994110a3b4d93bca7fadcaebafdc5 (127.30.218.3:41919): Not found: Tablet not found: 1e65f12167a346a39018672ca5370d74
I20260806 10:57:08.799486 14489 leader_election.cc:302] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d2c5277ea7546a3bbdef1d58c93fba2; no voters: a2291f6829ab46b084f98b6f7e974db4, ae6994110a3b4d93bca7fadcaebafdc5
I20260806 10:57:08.799741 14920 raft_consensus.cc:2726] T 1e65f12167a346a39018672ca5370d74 P 3d2c5277ea7546a3bbdef1d58c93fba2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 10:57:08.968963 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:08.968105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.3:37437)
0806 10:57:08.968677 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:08.968780 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":285,"threads_started":1}
I20260806 10:57:08.988406 14922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:08.987450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.1:51867)
0806 10:57:08.988050 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:08.988159 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":300,"threads_started":1}
I20260806 10:57:09.473897 14923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:09.473098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.2:40241)
0806 10:57:09.473619 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:09.473712 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":243,"threads_started":1}
I20260806 10:57:09.970993 14924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:09.970059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.3:53063)
0806 10:57:09.970659 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:09.970769 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":322,"threads_started":1}
I20260806 10:57:09.990504 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:09.989486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.1:34195)
0806 10:57:09.990138 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:09.990254 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":382,"threads_started":1}
I20260806 10:57:10.475659 14926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:10.474845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.2:37847)
0806 10:57:10.475347 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:10.475459 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":245,"threads_started":1}
I20260806 10:57:10.973155 14927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:10.972233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.3:47081)
0806 10:57:10.972804 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:10.972939 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":294,"threads_started":1}
I20260806 10:57:10.992548 14928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:10.991598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.1:41455)
0806 10:57:10.992177 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:10.992287 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":313,"threads_started":1}
I20260806 10:57:11.477900 14929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:11.476704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.2:48861)
0806 10:57:11.477504 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:11.477636 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":326,"threads_started":1}
I20260806 10:57:11.975275 14930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:11.974167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.3:48797)
0806 10:57:11.974865 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:11.975009 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":323,"threads_started":1}
I20260806 10:57:11.995340 14931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:11.994323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.1:53941)
0806 10:57:11.994954 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:11.995093 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":290,"threads_started":1}
I20260806 10:57:12.480428 14932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:12.479224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.2:36895)
0806 10:57:12.479948 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:12.480205 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":348,"threads_started":1}
I20260806 10:57:12.952724 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 14435
I20260806 10:57:12.977936 14934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:12.976897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.3:43093)
0806 10:57:12.977585 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:12.977733 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":316,"threads_started":1}
I20260806 10:57:12.997720 14935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:12.996600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.1:41969)
0806 10:57:12.997302 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:12.997430 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":348,"threads_started":1}
I20260806 10:57:13.482633 14936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:13.481589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:42721 (local address 127.30.218.2:39593)
0806 10:57:13.482284 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:13.482413 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":344,"threads_started":1}
I20260806 10:57:13.590128 14933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:12.954291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50490 (local address 127.30.218.1:43791)
0806 10:57:12.955516 (+ 1225us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:12.955526 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:13.570470 (+614944us) server_negotiation.cc:407] Connection header received
0806 10:57:13.570654 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:13.570662 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:13.570941 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:13.571118 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:57:13.572779 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:13.573876 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:13.575296 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:13.575897 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:13.576025 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:13.583865 (+ 7840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:57:13.583884 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:57:13.583902 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:57:13.583949 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:57:13.586900 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:13.587766 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:13.587774 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:13.587777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:13.587929 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:13.588327 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:13.588331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:13.588333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:13.588747 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:57:13.589081 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:13.589407 (+ 326us) server_negotiation.cc:299] Negotiation successful
0806 10:57:13.589739 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":992,"thread_start_us":802,"threads_started":1}
I20260806 10:57:13.591667 14600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:57:13.687631 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 14435
I20260806 10:57:13.709744 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 14609
I20260806 10:57:13.724968 14938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:13.710582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43846 (local address 127.30.218.2:33851)
0806 10:57:13.711234 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:13.711239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:13.711260 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:57:13.711341 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:13.711345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:13.711425 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:13.711535 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:57:13.713008 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:13.714104 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:13.715389 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:13.715883 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:13.715988 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:13.718757 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:57:13.718762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:57:13.718765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:57:13.718818 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:57:13.721992 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:13.722950 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:13.722954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:13.722957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:13.723110 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:13.723465 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:13.723467 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:13.723470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:13.723771 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:57:13.724026 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:13.724334 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 10:57:13.724538 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":321,"threads_started":1}
I20260806 10:57:13.725962 14736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:57:13.814442 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 14609
I20260806 10:57:13.833091 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 14744
I20260806 10:57:13.847523 14940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:13.834133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59676 (local address 127.30.218.3:41919)
0806 10:57:13.834760 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:13.834765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:13.834783 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:57:13.834847 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:13.834851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:13.834928 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:13.835032 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:57:13.836486 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:13.837321 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:13.838761 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:13.839162 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:13.839276 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:13.842077 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:57:13.842083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:57:13.842086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:57:13.842119 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:57:13.844701 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:13.845408 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:13.845413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:13.845415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:13.845598 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:13.846001 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:13.846006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:13.846008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:13.846311 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:57:13.846618 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:13.846887 (+ 269us) server_negotiation.cc:299] Negotiation successful
0806 10:57:13.847139 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":327,"threads_started":1}
I20260806 10:57:13.848352 14869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:57:13.936125 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 14744
2026-08-06T10:57:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25731 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:57:14 dist-test-slave-49h3 krb5kdc[14944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:57:14 dist-test-slave-49h3 krb5kdc[14944](info): set up 2 sockets
Aug 06 10:57:14 dist-test-slave-49h3 krb5kdc[14944](info): commencing operation
krb5kdc: starting...
W20260806 10:57:15.027055 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.005s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:57:15 dist-test-slave-49h3 krb5kdc[14944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:57:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:57:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:57:15.353766 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:46571
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:40935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:46571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:57:15.704146 14957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:57:15.704519 14957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:57:15.704676 14957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:57:15.712401 14957 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 10:57:15.712545 14957 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 10:57:15.712604 14957 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 10:57:15.712692 14957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:57:15.712764 14957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:57:15.712862 14957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:57:15.712926 14957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:57:15.713001 14957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:57:15.723503 14957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:40935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:46571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:46571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:57:15.724918 14957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:57:15.726446 14957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:57:15.738965 14965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:57:15.738997 14963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:57:15.739018 14962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:57:15.739399 14957 server_base.cc:1034] running on GCE node
I20260806 10:57:15.740490 14957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:57:15.742129 14957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:57:15.743359 14957 hybrid_clock.cc:648] HybridClock initialized: now 1786013835743281 us; error 63 us; skew 500 ppm
I20260806 10:57:15.743738 14957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:57:15 dist-test-slave-49h3 krb5kdc[14944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013835, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:57:15.747095 14957 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:57:15.748724 14957 webserver.cc:460] Webserver started at http://127.30.218.62:43097/ using document root <none> and password file <none>
I20260806 10:57:15.749509 14957 fs_manager.cc:359] Metadata directory not provided
I20260806 10:57:15.749634 14957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:57:15.749965 14957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:57:15.753278 14957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "1ecd9ddddd834083b02461abaaf1fef0"
format_stamp: "Formatted at 2026-08-06 10:57:15 on dist-test-slave-49h3"
server_key: "4879fbb8790a0c277cb72cfeb48a5a99"
server_key_iv: "c66071c2053ca53c73563e975acf081e"
server_key_version: "encryptionkey@0"
I20260806 10:57:15.754212 14957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "1ecd9ddddd834083b02461abaaf1fef0"
format_stamp: "Formatted at 2026-08-06 10:57:15 on dist-test-slave-49h3"
server_key: "4879fbb8790a0c277cb72cfeb48a5a99"
server_key_iv: "c66071c2053ca53c73563e975acf081e"
server_key_version: "encryptionkey@0"
I20260806 10:57:15.759910 14957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:57:15.763640 14973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:57:15.765075 14957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:57:15.765262 14957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "1ecd9ddddd834083b02461abaaf1fef0"
format_stamp: "Formatted at 2026-08-06 10:57:15 on dist-test-slave-49h3"
server_key: "4879fbb8790a0c277cb72cfeb48a5a99"
server_key_iv: "c66071c2053ca53c73563e975acf081e"
server_key_version: "encryptionkey@0"
I20260806 10:57:15.765520 14957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:57:15.781010 14957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:57:15.785480 14957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:57:15.785727 14957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:57:15.805941 14957 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:46571
I20260806 10:57:15.805965 15024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:46571 every 8 connection(s)
I20260806 10:57:15.807569 14957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:57:15.811476 15025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:57:15.815088 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 14957
I20260806 10:57:15.815281 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:57:15.816035 31592 external_mini_cluster.cc:1428] Setting key 6253d1925320260d569d06d49ea070b3
I20260806 10:57:15.824429 15025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0: Bootstrap starting.
I20260806 10:57:15.827740 15025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0: Neither blocks nor log segments found. Creating new log.
I20260806 10:57:15.828845 15025 log.cc:824] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0: Log is configured to *not* fsync() on all Append() calls
Aug 06 10:57:15 dist-test-slave-49h3 krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:57:15.831985 15025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0: No bootstrap required, opened a new log
I20260806 10:57:15.839205 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:15.818218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44712 (local address 127.30.218.62:46571)
0806 10:57:15.819065 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:15.819093 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:15.819137 (+ 44us) server_negotiation.cc:407] Connection header received
0806 10:57:15.820811 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:15.820838 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:15.821299 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:15.821737 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:57:15.823341 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:15.824712 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:15.826162 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:15.826875 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:15.827051 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:15.831279 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:57:15.831287 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:57:15.831303 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:57:15.831342 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:57:15.834565 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:15.835348 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:15.835356 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:15.835367 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:15.835562 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:15.835999 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:15.836003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:15.836005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:15.836660 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:57:15.836987 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:15.837541 (+ 554us) server_negotiation.cc:299] Negotiation successful
0806 10:57:15.838026 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":388,"threads_started":1}
I20260806 10:57:15.840529 15025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecd9ddddd834083b02461abaaf1fef0" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46571 } }
I20260806 10:57:15.841006 15025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:57:15.841120 15025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ecd9ddddd834083b02461abaaf1fef0, State: Initialized, Role: FOLLOWER
I20260806 10:57:15.841730 15025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecd9ddddd834083b02461abaaf1fef0" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46571 } }
I20260806 10:57:15.841995 15025 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:57:15.842139 15025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:57:15.842301 15025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:57:15.844707 15025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecd9ddddd834083b02461abaaf1fef0" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46571 } }
I20260806 10:57:15.845255 15025 leader_election.cc:302] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ecd9ddddd834083b02461abaaf1fef0; no voters:
I20260806 10:57:15.845906 15025 leader_election.cc:288] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:57:15.846057 15030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:57:15.846657 15030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 1 LEADER]: Becoming Leader. State: Replica: 1ecd9ddddd834083b02461abaaf1fef0, State: Running, Role: LEADER
I20260806 10:57:15.847373 15030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecd9ddddd834083b02461abaaf1fef0" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46571 } }
I20260806 10:57:15.848476 15025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:57:15.850456 15031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ecd9ddddd834083b02461abaaf1fef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecd9ddddd834083b02461abaaf1fef0" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46571 } } }
I20260806 10:57:15.850355 15032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ecd9ddddd834083b02461abaaf1fef0. Latest consensus state: current_term: 1 leader_uuid: "1ecd9ddddd834083b02461abaaf1fef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecd9ddddd834083b02461abaaf1fef0" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46571 } } }
I20260806 10:57:15.850823 15031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [sys.catalog]: This master's current role is: LEADER
I20260806 10:57:15.850823 15032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [sys.catalog]: This master's current role is: LEADER
I20260806 10:57:15.851758 15036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:57:15.857599 15036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:57:15.867895 15036 catalog_manager.cc:1347] Generated new cluster ID: 5b98922264d74c4d9a577c8a7c7444b5
I20260806 10:57:15.868028 15036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:57:15.879386 15036 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:57:15.880856 15036 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:57:15.891528 15036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0: Generated new TSK 0
I20260806 10:57:15.892473 15036 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
I20260806 10:57:15.963094 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:46571
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:40935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:57:16.313099 15053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:57:16.313413 15053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:57:16.313570 15053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:57:16.321098 15053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:57:16.321242 15053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:57:16.321403 15053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:57:16.331719 15053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:40935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:46571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:57:16.333076 15053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:57:16.334472 15053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:57:16.347911 15058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:57:16.347993 15059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:57:16.348016 15061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:57:16.349474 15053 server_base.cc:1034] running on GCE node
I20260806 10:57:16.350198 15053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:57:16.351246 15053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:57:16.352476 15053 hybrid_clock.cc:648] HybridClock initialized: now 1786013836352415 us; error 60 us; skew 500 ppm
I20260806 10:57:16.352811 15053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:57:16 dist-test-slave-49h3 krb5kdc[14944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013836, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:57:16.356041 15053 init.cc:377] Logged in from keytab as kudu/127.30.218.1@KRBTEST.COM (short username kudu)
I20260806 10:57:16.357482 15053 webserver.cc:460] Webserver started at http://127.30.218.1:44377/ using document root <none> and password file <none>
I20260806 10:57:16.358132 15053 fs_manager.cc:359] Metadata directory not provided
I20260806 10:57:16.358235 15053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:57:16.358496 15053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:57:16.361497 15053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "4405ef6882ce4212bda93cdaed80d15f"
format_stamp: "Formatted at 2026-08-06 10:57:16 on dist-test-slave-49h3"
server_key: "63d4a353fadf48ced7062d43554e30da"
server_key_iv: "a5593b2145c040cc727567c411cd0a40"
server_key_version: "encryptionkey@0"
I20260806 10:57:16.362269 15053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "4405ef6882ce4212bda93cdaed80d15f"
format_stamp: "Formatted at 2026-08-06 10:57:16 on dist-test-slave-49h3"
server_key: "63d4a353fadf48ced7062d43554e30da"
server_key_iv: "a5593b2145c040cc727567c411cd0a40"
server_key_version: "encryptionkey@0"
I20260806 10:57:16.367269 15053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260806 10:57:16.370797 15068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:57:16.372254 15053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:57:16.372435 15053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "4405ef6882ce4212bda93cdaed80d15f"
format_stamp: "Formatted at 2026-08-06 10:57:16 on dist-test-slave-49h3"
server_key: "63d4a353fadf48ced7062d43554e30da"
server_key_iv: "a5593b2145c040cc727567c411cd0a40"
server_key_version: "encryptionkey@0"
I20260806 10:57:16.372655 15053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:57:16.387264 15053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:57:16.391960 15053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:57:16.392275 15053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:57:16.393302 15053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:57:16.394730 15053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:57:16.394807 15053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:57:16.394928 15053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:57:16.394999 15053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:57:16.426478 15053 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:33447
I20260806 10:57:16.426522 15185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:33447 every 8 connection(s)
I20260806 10:57:16.427790 15053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:57:16.433955 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 15053
I20260806 10:57:16.434213 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:57:16.435051 31592 external_mini_cluster.cc:1428] Setting key 49fe8979d0f562e4fd2c07697f641af0
Aug 06 10:57:16 dist-test-slave-49h3 krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013836, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:57:16.450923 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:16.430471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:60517 (local address 127.30.218.62:46571)
0806 10:57:16.430941 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:16.430945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:16.431914 (+ 969us) server_negotiation.cc:407] Connection header received
0806 10:57:16.434088 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:16.434094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:16.434194 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:16.434315 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:57:16.436369 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:16.437267 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:16.439568 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:16.440013 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:16.440124 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:16.444523 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:57:16.444530 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:57:16.444533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:57:16.444581 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:57:16.447301 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:16.448321 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:16.448325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:16.448328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:16.448457 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:16.448980 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:16.448982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:16.448984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:16.449319 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:57:16.449484 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:16.450425 (+ 941us) server_negotiation.cc:299] Negotiation successful
0806 10:57:16.450646 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":234,"threads_started":1}
I20260806 10:57:16.452142 15190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:16.431011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.1:60517)
0806 10:57:16.431747 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:16.431786 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 10:57:16.433719 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:57:16.434619 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:57:16.434633 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:57:16.435468 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:57:16.436203 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:16.436223 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:16.437435 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:16.437440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:16.439379 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:16.439397 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:16.440269 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:16.440275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:16.440556 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:16.441303 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:57:16.441330 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:57:16.444188 (+ 2858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:57:16.447505 (+ 3317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:57:16.447517 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:57:16.447538 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:57:16.448198 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:57:16.448580 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:57:16.448584 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:57:16.448587 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:57:16.448869 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:57:16.449484 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:57:16.449492 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:57:16.449985 (+ 493us) client_negotiation.cc:769] Sending connection context
0806 10:57:16.450304 (+ 319us) client_negotiation.cc:240] Negotiation successful
0806 10:57:16.450877 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":294,"threads_started":1}
I20260806 10:57:16.453948 15186 heartbeater.cc:347] Connected to a master server at 127.30.218.62:46571
I20260806 10:57:16.454345 15186 heartbeater.cc:464] Registering TS with master...
I20260806 10:57:16.455174 15186 heartbeater.cc:511] Master 127.30.218.62:46571 requested a full tablet report, sending...
I20260806 10:57:16.457736 14990 ts_manager.cc:194] Registered new tserver with Master: 4405ef6882ce4212bda93cdaed80d15f (127.30.218.1:33447)
I20260806 10:57:16.459823 14990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.1@KRBTEST.COM'} at 127.30.218.1:60517
WARNING: no policy specified for kudu/127.30.218.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
I20260806 10:57:16.506910 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:46571
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:40935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:57:16.874441 15193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:57:16.874769 15193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:57:16.874903 15193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:57:16.882670 15193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:57:16.882812 15193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:57:16.882956 15193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:57:16.893517 15193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:40935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:46571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:57:16.894924 15193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:57:16.896356 15193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:57:16.910432 15198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:57:16.910477 15201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:57:16.910504 15199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:57:16.911356 15193 server_base.cc:1034] running on GCE node
I20260806 10:57:16.911989 15193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:57:16.913002 15193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:57:16.914214 15193 hybrid_clock.cc:648] HybridClock initialized: now 1786013836914151 us; error 56 us; skew 500 ppm
I20260806 10:57:16.914539 15193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:57:16 dist-test-slave-49h3 krb5kdc[14944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013836, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:57:16.917711 15193 init.cc:377] Logged in from keytab as kudu/127.30.218.2@KRBTEST.COM (short username kudu)
I20260806 10:57:16.919272 15193 webserver.cc:460] Webserver started at http://127.30.218.2:33919/ using document root <none> and password file <none>
I20260806 10:57:16.919966 15193 fs_manager.cc:359] Metadata directory not provided
I20260806 10:57:16.920069 15193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:57:16.920377 15193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:57:16.923353 15193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "028b0d0d65d2450e8a3725c99cc5ccbb"
format_stamp: "Formatted at 2026-08-06 10:57:16 on dist-test-slave-49h3"
server_key: "1996d0ace5792c489b29cb157e3f09e5"
server_key_iv: "0538d1575046b4c88b2ab5d61a152592"
server_key_version: "encryptionkey@0"
I20260806 10:57:16.924075 15193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "028b0d0d65d2450e8a3725c99cc5ccbb"
format_stamp: "Formatted at 2026-08-06 10:57:16 on dist-test-slave-49h3"
server_key: "1996d0ace5792c489b29cb157e3f09e5"
server_key_iv: "0538d1575046b4c88b2ab5d61a152592"
server_key_version: "encryptionkey@0"
I20260806 10:57:16.929172 15193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 10:57:16.932905 15208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:57:16.934372 15193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:57:16.934523 15193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "028b0d0d65d2450e8a3725c99cc5ccbb"
format_stamp: "Formatted at 2026-08-06 10:57:16 on dist-test-slave-49h3"
server_key: "1996d0ace5792c489b29cb157e3f09e5"
server_key_iv: "0538d1575046b4c88b2ab5d61a152592"
server_key_version: "encryptionkey@0"
I20260806 10:57:16.934719 15193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:57:16.949554 15193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:57:16.954200 15193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:57:16.954545 15193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:57:16.955605 15193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:57:16.957055 15193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:57:16.957136 15193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:57:16.957259 15193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:57:16.957319 15193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:57:16.987534 15193 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:43451
I20260806 10:57:16.987576 15323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:43451 every 8 connection(s)
I20260806 10:57:16.988878 15193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:57:16.989622 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 15193
I20260806 10:57:16.989845 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:57:16.990471 31592 external_mini_cluster.cc:1428] Setting key 33bcfa86cf530662b103e13f541523cf
Aug 06 10:57:16 dist-test-slave-49h3 krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013836, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:57:17.008862 15326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:16.991629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:48285 (local address 127.30.218.62:46571)
0806 10:57:16.992113 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:16.992120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:16.993162 (+ 1042us) server_negotiation.cc:407] Connection header received
0806 10:57:16.995080 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:16.995086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:16.995169 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:16.995271 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:57:16.996857 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:16.997633 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:16.999170 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:16.999606 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:16.999718 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:17.003260 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:57:17.003264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:57:17.003268 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:57:17.003300 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:57:17.006038 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:17.006693 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:17.006698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:17.006701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:17.006829 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:17.007226 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:17.007230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:17.007232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:17.007481 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:57:17.007627 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:17.008280 (+ 653us) server_negotiation.cc:299] Negotiation successful
0806 10:57:17.008497 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":236,"threads_started":1}
I20260806 10:57:17.009830 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:16.992275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.2:48285)
0806 10:57:16.993014 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:16.993053 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 10:57:16.994802 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:57:16.995523 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:57:16.995532 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:57:16.996098 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:57:16.996672 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:16.996689 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:16.997782 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:16.997787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:16.999027 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:16.999039 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:16.999849 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:16.999857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:17.000082 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:17.000740 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:57:17.000763 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:57:17.003038 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:57:17.006168 (+ 3130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:57:17.006177 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:57:17.006192 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:57:17.006582 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:57:17.006929 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:57:17.006934 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:57:17.006938 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:57:17.007115 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:57:17.007611 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:57:17.007618 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:57:17.007932 (+ 314us) client_negotiation.cc:769] Sending connection context
0806 10:57:17.008162 (+ 230us) client_negotiation.cc:240] Negotiation successful
0806 10:57:17.008663 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":315,"threads_started":1}
I20260806 10:57:17.011296 15324 heartbeater.cc:347] Connected to a master server at 127.30.218.62:46571
I20260806 10:57:17.011582 15324 heartbeater.cc:464] Registering TS with master...
I20260806 10:57:17.012281 15324 heartbeater.cc:511] Master 127.30.218.62:46571 requested a full tablet report, sending...
I20260806 10:57:17.013808 14990 ts_manager.cc:194] Registered new tserver with Master: 028b0d0d65d2450e8a3725c99cc5ccbb (127.30.218.2:43451)
I20260806 10:57:17.014894 14990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.2@KRBTEST.COM'} at 127.30.218.2:48285
WARNING: no policy specified for kudu/127.30.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
I20260806 10:57:17.056278 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:46571
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:40935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:57:17.401542 15330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:57:17.401837 15330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:57:17.401991 15330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:57:17.409286 15330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:57:17.409417 15330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:57:17.409593 15330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:57:17.419931 15330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:40935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:46571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:57:17.421334 15330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:57:17.422724 15330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:57:17.437863 15339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:57:17.438009 15336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:57:17.437958 15337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:57:17.439014 15330 server_base.cc:1034] running on GCE node
I20260806 10:57:17.439755 15330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:57:17.440879 15330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:57:17.442114 15330 hybrid_clock.cc:648] HybridClock initialized: now 1786013837442051 us; error 64 us; skew 500 ppm
I20260806 10:57:17.442472 15330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:57:17 dist-test-slave-49h3 krb5kdc[14944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013837, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:57:17.445906 15330 init.cc:377] Logged in from keytab as kudu/127.30.218.3@KRBTEST.COM (short username kudu)
I20260806 10:57:17.447638 15330 webserver.cc:460] Webserver started at http://127.30.218.3:42639/ using document root <none> and password file <none>
I20260806 10:57:17.448370 15330 fs_manager.cc:359] Metadata directory not provided
I20260806 10:57:17.448498 15330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:57:17.448835 15330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:57:17.452062 15330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "e42dd462ddfe4d979ee3662dc9f29bae"
format_stamp: "Formatted at 2026-08-06 10:57:17 on dist-test-slave-49h3"
server_key: "b1f97602c1a91ee92dc89f7941771bda"
server_key_iv: "72132c406671b929f6570c1acd6941d0"
server_key_version: "encryptionkey@0"
I20260806 10:57:17.452864 15330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "e42dd462ddfe4d979ee3662dc9f29bae"
format_stamp: "Formatted at 2026-08-06 10:57:17 on dist-test-slave-49h3"
server_key: "b1f97602c1a91ee92dc89f7941771bda"
server_key_iv: "72132c406671b929f6570c1acd6941d0"
server_key_version: "encryptionkey@0"
I20260806 10:57:17.458710 15330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 10:57:17.462564 15346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:57:17.463439 15186 heartbeater.cc:503] Master 127.30.218.62:46571 was elected leader, sending a full tablet report...
I20260806 10:57:17.464275 15330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:57:17.464455 15330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "e42dd462ddfe4d979ee3662dc9f29bae"
format_stamp: "Formatted at 2026-08-06 10:57:17 on dist-test-slave-49h3"
server_key: "b1f97602c1a91ee92dc89f7941771bda"
server_key_iv: "72132c406671b929f6570c1acd6941d0"
server_key_version: "encryptionkey@0"
I20260806 10:57:17.464700 15330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:57:17.480758 15330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:57:17.484946 15330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:57:17.485235 15330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:57:17.486248 15330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:57:17.487617 15330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:57:17.487697 15330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:57:17.487819 15330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:57:17.487874 15330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:57:17.517944 15330 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:35047
I20260806 10:57:17.518013 15461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:35047 every 8 connection(s)
I20260806 10:57:17.519217 15330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:57:17.527381 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 15330
I20260806 10:57:17.527578 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:57:17.528203 31592 external_mini_cluster.cc:1428] Setting key 9bd35c28eb8334c307e2b5536b5d31f0
Aug 06 10:57:17 dist-test-slave-49h3 krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013837, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:57:17.539781 15464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:17.521639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:35049 (local address 127.30.218.62:46571)
0806 10:57:17.522142 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:17.522148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:17.523076 (+ 928us) server_negotiation.cc:407] Connection header received
0806 10:57:17.524896 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:17.524900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:17.524974 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:17.525075 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:57:17.526625 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.527463 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.529489 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.529913 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.530031 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:17.533743 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:57:17.533748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:57:17.533751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:57:17.533789 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:57:17.536239 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:17.537170 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:17.537175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:17.537177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:17.537315 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:17.537822 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:17.537826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:17.537828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:17.538138 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:57:17.538290 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:17.539120 (+ 830us) server_negotiation.cc:299] Negotiation successful
0806 10:57:17.539372 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":218,"threads_started":1}
I20260806 10:57:17.540827 15465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:17.522166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.3:35049)
0806 10:57:17.522912 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:17.522950 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 10:57:17.524604 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:57:17.525267 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:57:17.525274 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:57:17.525886 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:57:17.526467 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:17.526485 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.527643 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.527648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:17.529305 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:17.529319 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.530163 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.530171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:17.530402 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:17.531206 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:57:17.531238 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:57:17.533500 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:57:17.536405 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:57:17.536416 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:57:17.536434 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:57:17.537034 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:57:17.537427 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:57:17.537432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:57:17.537436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:57:17.537696 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:57:17.538270 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:57:17.538279 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:57:17.538688 (+ 409us) client_negotiation.cc:769] Sending connection context
0806 10:57:17.538958 (+ 270us) client_negotiation.cc:240] Negotiation successful
0806 10:57:17.539550 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":298,"threads_started":1}
I20260806 10:57:17.542436 15462 heartbeater.cc:347] Connected to a master server at 127.30.218.62:46571
I20260806 10:57:17.542762 15462 heartbeater.cc:464] Registering TS with master...
I20260806 10:57:17.543665 15462 heartbeater.cc:511] Master 127.30.218.62:46571 requested a full tablet report, sending...
I20260806 10:57:17.545136 14990 ts_manager.cc:194] Registered new tserver with Master: e42dd462ddfe4d979ee3662dc9f29bae (127.30.218.3:35047)
I20260806 10:57:17.546041 14990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.3@KRBTEST.COM'} at 127.30.218.3:35049
I20260806 10:57:17.556349 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:57:17.574669 15464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:17.561326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44720 (local address 127.30.218.62:46571)
0806 10:57:17.561544 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:17.561550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:17.561932 (+ 382us) server_negotiation.cc:407] Connection header received
0806 10:57:17.562101 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:17.562105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:17.562175 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:17.562266 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:57:17.563698 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.564473 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.566000 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.566483 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.566586 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:17.569391 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:57:17.569396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:57:17.569399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:57:17.569439 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:57:17.572094 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:17.572904 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:17.572907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:17.572910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:17.573039 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:17.573364 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:17.573367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:17.573369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:17.573740 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:57:17.573936 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:17.574228 (+ 292us) server_negotiation.cc:299] Negotiation successful
0806 10:57:17.574423 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 10:57:17.579093 14990 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:57:17.586529 14990 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:57:17.624183 15475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:17.599835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:33447 (local address 127.0.0.1:53560)
0806 10:57:17.601387 (+ 1552us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:17.601441 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 10:57:17.601691 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:57:17.617332 (+ 15641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:57:17.617337 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:57:17.617367 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:57:17.617815 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:17.617828 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.619922 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.619925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:17.621574 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:17.621589 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.623205 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.623212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:17.623418 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:17.623477 (+ 59us) client_negotiation.cc:769] Sending connection context
0806 10:57:17.623578 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 10:57:17.623745 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1254,"mutex_wait_us":3,"thread_start_us":1065,"threads_started":1}
I20260806 10:57:17.624449 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:17.602094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55338 (local address 127.30.218.2:43451)
0806 10:57:17.614982 (+ 12888us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:17.614992 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:17.616041 (+ 1049us) server_negotiation.cc:407] Connection header received
0806 10:57:17.616140 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:17.616147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:17.616340 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:17.616500 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:57:17.617548 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.618744 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.621562 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.623020 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.623175 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:17.623407 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:17.623982 (+ 575us) server_negotiation.cc:299] Negotiation successful
0806 10:57:17.624171 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12688,"thread_start_us":12564,"threads_started":1}
I20260806 10:57:17.624948 15478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:17.601120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57154 (local address 127.30.218.3:35047)
0806 10:57:17.608856 (+ 7736us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:17.608868 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:17.615612 (+ 6744us) server_negotiation.cc:407] Connection header received
0806 10:57:17.615873 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:17.615882 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:17.616220 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:17.616488 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:57:17.617411 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.619274 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.621875 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.623244 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.623355 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:17.623569 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:17.624256 (+ 687us) server_negotiation.cc:299] Negotiation successful
0806 10:57:17.624610 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7531,"thread_start_us":7380,"threads_started":1}
I20260806 10:57:17.625137 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:17.600450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:35047 (local address 127.0.0.1:57154)
0806 10:57:17.615440 (+ 14990us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:17.615468 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 10:57:17.615680 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:57:17.616530 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:57:17.616538 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:57:17.616572 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:57:17.617137 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:17.617153 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.619429 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.619434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:17.621704 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:17.621719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.623708 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.623713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:17.623877 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:17.623905 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 10:57:17.623973 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 10:57:17.624086 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14816,"mutex_wait_us":81,"thread_start_us":14640,"threads_started":1}
I20260806 10:57:17.625600 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:17.601888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:43451 (local address 127.0.0.1:55338)
0806 10:57:17.615765 (+ 13877us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:17.615783 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:57:17.615923 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:57:17.616776 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:57:17.616781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:57:17.616803 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:57:17.617311 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:17.617324 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.618884 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.618888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:17.621354 (+ 2466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:17.621371 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.623569 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.623576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:17.623776 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:17.623809 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 10:57:17.623907 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 10:57:17.624062 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13762,"mutex_wait_us":86,"thread_start_us":13597,"threads_started":1}
I20260806 10:57:17.625919 15476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:17.600177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53560 (local address 127.30.218.1:33447)
0806 10:57:17.616306 (+ 16129us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:17.616312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:17.616335 (+ 23us) server_negotiation.cc:407] Connection header received
0806 10:57:17.616416 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:17.616422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:17.617106 (+ 684us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:17.618147 (+ 1041us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:57:17.618460 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.619786 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.621750 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.622738 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.625128 (+ 2390us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:17.625382 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:17.625465 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 10:57:17.625609 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15969,"thread_start_us":15805,"threads_started":1}
I20260806 10:57:17.628897 15395 tablet_service.cc:1467] Processing CreateTablet for tablet 1ced271fa5ba49bba6a91935178ed45f (DEFAULT_TABLE table=test-table [id=d2b0e2884de24ab5a08bdd64ef38271d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:57:17.629166 15118 tablet_service.cc:1467] Processing CreateTablet for tablet 1ced271fa5ba49bba6a91935178ed45f (DEFAULT_TABLE table=test-table [id=d2b0e2884de24ab5a08bdd64ef38271d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:57:17.629166 15258 tablet_service.cc:1467] Processing CreateTablet for tablet 1ced271fa5ba49bba6a91935178ed45f (DEFAULT_TABLE table=test-table [id=d2b0e2884de24ab5a08bdd64ef38271d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:57:17.630285 15395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ced271fa5ba49bba6a91935178ed45f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:57:17.630558 15258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ced271fa5ba49bba6a91935178ed45f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:57:17.630584 15118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ced271fa5ba49bba6a91935178ed45f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:57:17.643989 15481 tablet_bootstrap.cc:492] T 1ced271fa5ba49bba6a91935178ed45f P e42dd462ddfe4d979ee3662dc9f29bae: Bootstrap starting.
I20260806 10:57:17.646368 15482 tablet_bootstrap.cc:492] T 1ced271fa5ba49bba6a91935178ed45f P 028b0d0d65d2450e8a3725c99cc5ccbb: Bootstrap starting.
I20260806 10:57:17.647217 15484 tablet_bootstrap.cc:492] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f: Bootstrap starting.
I20260806 10:57:17.648001 15481 tablet_bootstrap.cc:654] T 1ced271fa5ba49bba6a91935178ed45f P e42dd462ddfe4d979ee3662dc9f29bae: Neither blocks nor log segments found. Creating new log.
I20260806 10:57:17.649374 15481 log.cc:824] T 1ced271fa5ba49bba6a91935178ed45f P e42dd462ddfe4d979ee3662dc9f29bae: Log is configured to *not* fsync() on all Append() calls
I20260806 10:57:17.650506 15482 tablet_bootstrap.cc:654] T 1ced271fa5ba49bba6a91935178ed45f P 028b0d0d65d2450e8a3725c99cc5ccbb: Neither blocks nor log segments found. Creating new log.
I20260806 10:57:17.650913 15484 tablet_bootstrap.cc:654] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f: Neither blocks nor log segments found. Creating new log.
I20260806 10:57:17.651749 15482 log.cc:824] T 1ced271fa5ba49bba6a91935178ed45f P 028b0d0d65d2450e8a3725c99cc5ccbb: Log is configured to *not* fsync() on all Append() calls
I20260806 10:57:17.652146 15484 log.cc:824] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f: Log is configured to *not* fsync() on all Append() calls
I20260806 10:57:17.653731 15481 tablet_bootstrap.cc:492] T 1ced271fa5ba49bba6a91935178ed45f P e42dd462ddfe4d979ee3662dc9f29bae: No bootstrap required, opened a new log
I20260806 10:57:17.654032 15481 ts_tablet_manager.cc:1397] T 1ced271fa5ba49bba6a91935178ed45f P e42dd462ddfe4d979ee3662dc9f29bae: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260806 10:57:17.655968 15484 tablet_bootstrap.cc:492] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f: No bootstrap required, opened a new log
I20260806 10:57:17.656225 15484 ts_tablet_manager.cc:1397] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260806 10:57:17.656240 15482 tablet_bootstrap.cc:492] T 1ced271fa5ba49bba6a91935178ed45f P 028b0d0d65d2450e8a3725c99cc5ccbb: No bootstrap required, opened a new log
I20260806 10:57:17.656483 15482 ts_tablet_manager.cc:1397] T 1ced271fa5ba49bba6a91935178ed45f P 028b0d0d65d2450e8a3725c99cc5ccbb: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260806 10:57:17.663558 15484 raft_consensus.cc:348] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4405ef6882ce4212bda93cdaed80d15f" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33447 } } peers { permanent_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 43451 } } peers { permanent_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35047 } }
I20260806 10:57:17.663975 15484 raft_consensus.cc:374] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:57:17.664072 15484 raft_consensus.cc:729] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4405ef6882ce4212bda93cdaed80d15f, State: Initialized, Role: FOLLOWER
I20260806 10:57:17.664703 15484 consensus_queue.cc:260] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4405ef6882ce4212bda93cdaed80d15f" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33447 } } peers { permanent_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 43451 } } peers { permanent_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35047 } }
I20260806 10:57:17.664667 15481 raft_consensus.cc:348] T 1ced271fa5ba49bba6a91935178ed45f P e42dd462ddfe4d979ee3662dc9f29bae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4405ef6882ce4212bda93cdaed80d15f" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33447 } } peers { permanent_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 43451 } } peers { permanent_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35047 } }
I20260806 10:57:17.665133 15481 raft_consensus.cc:374] T 1ced271fa5ba49bba6a91935178ed45f P e42dd462ddfe4d979ee3662dc9f29bae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:57:17.665228 15481 raft_consensus.cc:729] T 1ced271fa5ba49bba6a91935178ed45f P e42dd462ddfe4d979ee3662dc9f29bae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e42dd462ddfe4d979ee3662dc9f29bae, State: Initialized, Role: FOLLOWER
I20260806 10:57:17.665998 15481 consensus_queue.cc:260] T 1ced271fa5ba49bba6a91935178ed45f P e42dd462ddfe4d979ee3662dc9f29bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4405ef6882ce4212bda93cdaed80d15f" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33447 } } peers { permanent_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 43451 } } peers { permanent_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35047 } }
I20260806 10:57:17.666989 15482 raft_consensus.cc:348] T 1ced271fa5ba49bba6a91935178ed45f P 028b0d0d65d2450e8a3725c99cc5ccbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4405ef6882ce4212bda93cdaed80d15f" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33447 } } peers { permanent_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 43451 } } peers { permanent_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35047 } }
I20260806 10:57:17.667469 15482 raft_consensus.cc:374] T 1ced271fa5ba49bba6a91935178ed45f P 028b0d0d65d2450e8a3725c99cc5ccbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:57:17.667557 15482 raft_consensus.cc:729] T 1ced271fa5ba49bba6a91935178ed45f P 028b0d0d65d2450e8a3725c99cc5ccbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 028b0d0d65d2450e8a3725c99cc5ccbb, State: Initialized, Role: FOLLOWER
I20260806 10:57:17.667623 15462 heartbeater.cc:503] Master 127.30.218.62:46571 was elected leader, sending a full tablet report...
I20260806 10:57:17.668414 15481 ts_tablet_manager.cc:1428] T 1ced271fa5ba49bba6a91935178ed45f P e42dd462ddfe4d979ee3662dc9f29bae: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260806 10:57:17.668299 15482 consensus_queue.cc:260] T 1ced271fa5ba49bba6a91935178ed45f P 028b0d0d65d2450e8a3725c99cc5ccbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4405ef6882ce4212bda93cdaed80d15f" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33447 } } peers { permanent_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 43451 } } peers { permanent_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35047 } }
I20260806 10:57:17.670688 15484 ts_tablet_manager.cc:1428] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260806 10:57:17.672128 15324 heartbeater.cc:503] Master 127.30.218.62:46571 was elected leader, sending a full tablet report...
I20260806 10:57:17.672803 15482 ts_tablet_manager.cc:1428] T 1ced271fa5ba49bba6a91935178ed45f P 028b0d0d65d2450e8a3725c99cc5ccbb: Time spent starting tablet: real 0.016s user 0.008s sys 0.007s
W20260806 10:57:17.680176 15187 tablet.cc:2369] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:57:17.712340 15488 raft_consensus.cc:482] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:57:17.712687 15488 raft_consensus.cc:504] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4405ef6882ce4212bda93cdaed80d15f" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33447 } } peers { permanent_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 43451 } } peers { permanent_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35047 } }
I20260806 10:57:17.714813 15488 leader_election.cc:288] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 028b0d0d65d2450e8a3725c99cc5ccbb (127.30.218.2:43451), e42dd462ddfe4d979ee3662dc9f29bae (127.30.218.3:35047)
I20260806 10:57:17.723699 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:17.715343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:40627 (local address 127.30.218.2:43451)
0806 10:57:17.715845 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:17.715851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:17.716194 (+ 343us) server_negotiation.cc:407] Connection header received
0806 10:57:17.716426 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:17.716431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:17.716502 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:17.716591 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:57:17.717324 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.718480 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.721355 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.722611 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.722807 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:17.722873 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:17.723285 (+ 412us) server_negotiation.cc:299] Negotiation successful
0806 10:57:17.723420 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333}
I20260806 10:57:17.723699 15478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:17.715578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:33143 (local address 127.30.218.3:35047)
0806 10:57:17.715803 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:17.715809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:17.716259 (+ 450us) server_negotiation.cc:407] Connection header received
0806 10:57:17.716404 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:17.716409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:17.716495 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:17.716591 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:57:17.717353 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.718485 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.721355 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.722625 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.722793 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:17.722871 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:17.723282 (+ 411us) server_negotiation.cc:299] Negotiation successful
0806 10:57:17.723420 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 10:57:17.723912 15491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:17.715101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:35047 (local address 127.30.218.1:33143)
0806 10:57:17.716174 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:17.716189 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:57:17.716280 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:57:17.716625 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:57:17.716630 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:57:17.716732 (+ 102us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:57:17.717207 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:17.717216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.718649 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.718655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:17.721191 (+ 2536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:17.721209 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.722885 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.722893 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:17.723086 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:17.723147 (+ 61us) client_negotiation.cc:769] Sending connection context
0806 10:57:17.723255 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 10:57:17.723466 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"mutex_wait_us":89,"spinlock_wait_cycles":27520,"thread_start_us":348,"threads_started":1}
I20260806 10:57:17.724298 15492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:17.715078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:43451 (local address 127.30.218.1:40627)
0806 10:57:17.716070 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:17.716091 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:57:17.716272 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:57:17.716625 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:57:17.716630 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:57:17.716662 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:57:17.717167 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:17.717184 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.718649 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.718655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:17.721201 (+ 2546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:17.721210 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.722923 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.722929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:17.723158 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:17.723184 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 10:57:17.723255 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 10:57:17.723466 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":245,"thread_start_us":715,"threads_started":1}
I20260806 10:57:17.725049 15415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ced271fa5ba49bba6a91935178ed45f" candidate_uuid: "4405ef6882ce4212bda93cdaed80d15f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" is_pre_election: true
I20260806 10:57:17.725271 15278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ced271fa5ba49bba6a91935178ed45f" candidate_uuid: "4405ef6882ce4212bda93cdaed80d15f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb" is_pre_election: true
I20260806 10:57:17.725610 15415 raft_consensus.cc:2454] T 1ced271fa5ba49bba6a91935178ed45f P e42dd462ddfe4d979ee3662dc9f29bae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4405ef6882ce4212bda93cdaed80d15f in term 0.
I20260806 10:57:17.725780 15278 raft_consensus.cc:2454] T 1ced271fa5ba49bba6a91935178ed45f P 028b0d0d65d2450e8a3725c99cc5ccbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4405ef6882ce4212bda93cdaed80d15f in term 0.
I20260806 10:57:17.726361 15071 leader_election.cc:302] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4405ef6882ce4212bda93cdaed80d15f, e42dd462ddfe4d979ee3662dc9f29bae; no voters:
I20260806 10:57:17.726819 15488 raft_consensus.cc:2781] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:57:17.726987 15488 raft_consensus.cc:482] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:57:17.727110 15488 raft_consensus.cc:3037] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:57:17.729318 15488 raft_consensus.cc:504] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4405ef6882ce4212bda93cdaed80d15f" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33447 } } peers { permanent_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 43451 } } peers { permanent_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35047 } }
I20260806 10:57:17.730054 15488 leader_election.cc:288] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f [CANDIDATE]: Term 1 election: Requested vote from peers 028b0d0d65d2450e8a3725c99cc5ccbb (127.30.218.2:43451), e42dd462ddfe4d979ee3662dc9f29bae (127.30.218.3:35047)
I20260806 10:57:17.730618 15415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ced271fa5ba49bba6a91935178ed45f" candidate_uuid: "4405ef6882ce4212bda93cdaed80d15f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e42dd462ddfe4d979ee3662dc9f29bae"
I20260806 10:57:17.730603 15278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ced271fa5ba49bba6a91935178ed45f" candidate_uuid: "4405ef6882ce4212bda93cdaed80d15f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb"
I20260806 10:57:17.730845 15415 raft_consensus.cc:3037] T 1ced271fa5ba49bba6a91935178ed45f P e42dd462ddfe4d979ee3662dc9f29bae [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:57:17.730845 15278 raft_consensus.cc:3037] T 1ced271fa5ba49bba6a91935178ed45f P 028b0d0d65d2450e8a3725c99cc5ccbb [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:57:17.734269 15415 raft_consensus.cc:2454] T 1ced271fa5ba49bba6a91935178ed45f P e42dd462ddfe4d979ee3662dc9f29bae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4405ef6882ce4212bda93cdaed80d15f in term 1.
I20260806 10:57:17.734269 15278 raft_consensus.cc:2454] T 1ced271fa5ba49bba6a91935178ed45f P 028b0d0d65d2450e8a3725c99cc5ccbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4405ef6882ce4212bda93cdaed80d15f in term 1.
I20260806 10:57:17.734903 15070 leader_election.cc:302] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 028b0d0d65d2450e8a3725c99cc5ccbb, 4405ef6882ce4212bda93cdaed80d15f, e42dd462ddfe4d979ee3662dc9f29bae; no voters:
I20260806 10:57:17.735301 15488 raft_consensus.cc:2781] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:57:17.735769 15488 raft_consensus.cc:686] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f [term 1 LEADER]: Becoming Leader. State: Replica: 4405ef6882ce4212bda93cdaed80d15f, State: Running, Role: LEADER
I20260806 10:57:17.736286 15488 consensus_queue.cc:237] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4405ef6882ce4212bda93cdaed80d15f" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33447 } } peers { permanent_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 43451 } } peers { permanent_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35047 } }
W20260806 10:57:17.741154 15325 tablet.cc:2369] T 1ced271fa5ba49bba6a91935178ed45f P 028b0d0d65d2450e8a3725c99cc5ccbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:57:17.741132 14988 catalog_manager.cc:5495] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4405ef6882ce4212bda93cdaed80d15f (127.30.218.1). New cstate: current_term: 1 leader_uuid: "4405ef6882ce4212bda93cdaed80d15f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4405ef6882ce4212bda93cdaed80d15f" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 43451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35047 } health_report { overall_health: UNKNOWN } } }
I20260806 10:57:17.770102 15476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:17.764248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53564 (local address 127.30.218.1:33447)
0806 10:57:17.764424 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:17.764428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:17.764446 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:57:17.764575 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:17.764579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:17.764685 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:17.764812 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:57:17.765112 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.765874 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.767568 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.768039 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.768150 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:17.768927 (+ 777us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:57:17.769560 (+ 633us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:57:17.769711 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:17.769814 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 10:57:17.769892 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 10:57:17.770732 15463 tablet.cc:2369] T 1ced271fa5ba49bba6a91935178ed45f P e42dd462ddfe4d979ee3662dc9f29bae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:57:17.778934 15278 raft_consensus.cc:1261] T 1ced271fa5ba49bba6a91935178ed45f P 028b0d0d65d2450e8a3725c99cc5ccbb [term 1 FOLLOWER]: Refusing update from remote peer 4405ef6882ce4212bda93cdaed80d15f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:57:17.778934 15415 raft_consensus.cc:1261] T 1ced271fa5ba49bba6a91935178ed45f P e42dd462ddfe4d979ee3662dc9f29bae [term 1 FOLLOWER]: Refusing update from remote peer 4405ef6882ce4212bda93cdaed80d15f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:57:17.780195 15493 consensus_queue.cc:1035] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:57:17.780589 15488 consensus_queue.cc:1035] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f [LEADER]: Connected to new peer: Peer: permanent_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 43451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:57:17.797559 15497 mvcc.cc:205] Tried to move back new op lower bound from 7315512679531163648 to 7315512679372066816. Current Snapshot: MvccSnapshot[applied={T|T < 7315512679531163648}]
I20260806 10:57:17.798508 15496 mvcc.cc:205] Tried to move back new op lower bound from 7315512679531163648 to 7315512679372066816. Current Snapshot: MvccSnapshot[applied={T|T < 7315512679531163648}]
I20260806 10:57:17.814404 14989 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:57:17.814822 14989 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:57:17.820071 14989 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0: Sending DeleteTablet for 3 replicas of tablet 1ced271fa5ba49bba6a91935178ed45f
I20260806 10:57:17.821708 15395 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ced271fa5ba49bba6a91935178ed45f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:57:17 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:57154
I20260806 10:57:17.821961 15258 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ced271fa5ba49bba6a91935178ed45f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:57:17 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:55338
I20260806 10:57:17.822561 15118 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ced271fa5ba49bba6a91935178ed45f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:57:17 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:53560
I20260806 10:57:17.822746 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 15053
I20260806 10:57:17.834470 15507 tablet_replica.cc:330] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f: stopping tablet replica
I20260806 10:57:17.835108 15507 raft_consensus.cc:2229] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:57:17.836047 15507 raft_consensus.cc:2258] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:57:17.838713 15505 tablet_replica.cc:330] T 1ced271fa5ba49bba6a91935178ed45f P e42dd462ddfe4d979ee3662dc9f29bae: stopping tablet replica
I20260806 10:57:17.840893 15507 ts_tablet_manager.cc:1905] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:57:17.842005 15506 tablet_replica.cc:330] T 1ced271fa5ba49bba6a91935178ed45f P 028b0d0d65d2450e8a3725c99cc5ccbb: stopping tablet replica
I20260806 10:57:17.839330 15505 raft_consensus.cc:2229] T 1ced271fa5ba49bba6a91935178ed45f P e42dd462ddfe4d979ee3662dc9f29bae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:57:17.842514 15506 raft_consensus.cc:2229] T 1ced271fa5ba49bba6a91935178ed45f P 028b0d0d65d2450e8a3725c99cc5ccbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:57:17.842878 15505 raft_consensus.cc:2258] T 1ced271fa5ba49bba6a91935178ed45f P e42dd462ddfe4d979ee3662dc9f29bae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:57:17.843089 15506 raft_consensus.cc:2258] T 1ced271fa5ba49bba6a91935178ed45f P 028b0d0d65d2450e8a3725c99cc5ccbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:57:17.845618 15506 ts_tablet_manager.cc:1905] T 1ced271fa5ba49bba6a91935178ed45f P 028b0d0d65d2450e8a3725c99cc5ccbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:57:17.847087 15505 ts_tablet_manager.cc:1905] T 1ced271fa5ba49bba6a91935178ed45f P e42dd462ddfe4d979ee3662dc9f29bae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:57:17.850045 15507 ts_tablet_manager.cc:1918] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:57:17.850283 15507 log.cc:1196] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/wal/wals/1ced271fa5ba49bba6a91935178ed45f
I20260806 10:57:17.850889 15507 ts_tablet_manager.cc:1939] T 1ced271fa5ba49bba6a91935178ed45f P 4405ef6882ce4212bda93cdaed80d15f: Deleting consensus metadata
I20260806 10:57:17.853173 14975 catalog_manager.cc:4841] TS 4405ef6882ce4212bda93cdaed80d15f (127.30.218.1:33447): tablet 1ced271fa5ba49bba6a91935178ed45f (table test-table [id=d2b0e2884de24ab5a08bdd64ef38271d]) successfully deleted
I20260806 10:57:17.853508 15506 ts_tablet_manager.cc:1918] T 1ced271fa5ba49bba6a91935178ed45f P 028b0d0d65d2450e8a3725c99cc5ccbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Aug 06 10:57:17 dist-test-slave-49h3 krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.1@KRBTEST.COM
I20260806 10:57:17.853698 15506 log.cc:1196] T 1ced271fa5ba49bba6a91935178ed45f P 028b0d0d65d2450e8a3725c99cc5ccbb: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/1ced271fa5ba49bba6a91935178ed45f
I20260806 10:57:17.854250 15506 ts_tablet_manager.cc:1939] T 1ced271fa5ba49bba6a91935178ed45f P 028b0d0d65d2450e8a3725c99cc5ccbb: Deleting consensus metadata
I20260806 10:57:17.856442 14975 catalog_manager.cc:4841] TS 028b0d0d65d2450e8a3725c99cc5ccbb (127.30.218.2:43451): tablet 1ced271fa5ba49bba6a91935178ed45f (table test-table [id=d2b0e2884de24ab5a08bdd64ef38271d]) successfully deleted
I20260806 10:57:17.858914 15505 ts_tablet_manager.cc:1918] T 1ced271fa5ba49bba6a91935178ed45f P e42dd462ddfe4d979ee3662dc9f29bae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:57:17.859104 15505 log.cc:1196] T 1ced271fa5ba49bba6a91935178ed45f P e42dd462ddfe4d979ee3662dc9f29bae: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/1ced271fa5ba49bba6a91935178ed45f
I20260806 10:57:17.859560 15505 ts_tablet_manager.cc:1939] T 1ced271fa5ba49bba6a91935178ed45f P e42dd462ddfe4d979ee3662dc9f29bae: Deleting consensus metadata
I20260806 10:57:17.861302 14976 catalog_manager.cc:4841] TS e42dd462ddfe4d979ee3662dc9f29bae (127.30.218.3:35047): tablet 1ced271fa5ba49bba6a91935178ed45f (table test-table [id=d2b0e2884de24ab5a08bdd64ef38271d]) successfully deleted
I20260806 10:57:17.867262 15476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:17.824138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53570 (local address 127.30.218.1:33447)
0806 10:57:17.824344 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:17.824350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:17.834218 (+ 9868us) server_negotiation.cc:407] Connection header received
0806 10:57:17.834942 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:17.834949 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:17.835048 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:17.835137 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:57:17.839611 (+ 4474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.840814 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.844454 (+ 3640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.845214 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.845327 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:17.859202 (+ 13875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:57:17.859242 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:57:17.859263 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:57:17.859327 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:57:17.863779 (+ 4452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:17.864725 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:17.864736 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:17.864740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:17.864949 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:17.865368 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:17.865374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:17.865377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:17.865935 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:57:17.866268 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:17.866587 (+ 319us) server_negotiation.cc:299] Negotiation successful
0806 10:57:17.866871 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 10:57:17.868009 15181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:57:17.974197 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 15053
I20260806 10:57:17.993701 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 15193
Aug 06 10:57:17 dist-test-slave-49h3 krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.2@KRBTEST.COM
I20260806 10:57:18.010998 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:17.994510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55346 (local address 127.30.218.2:43451)
0806 10:57:17.994707 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:17.994713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:17.994735 (+ 22us) server_negotiation.cc:407] Connection header received
0806 10:57:17.994816 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:17.994822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:17.994905 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:17.994995 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:57:17.996228 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.997422 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.998737 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:17.999385 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:17.999512 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:18.003860 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:57:18.003874 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:57:18.003893 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:57:18.003948 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:57:18.008073 (+ 4125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:18.008914 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:18.008922 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:18.008925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:18.009060 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:18.009374 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:18.009377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:18.009379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:18.009855 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:57:18.010094 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:18.010462 (+ 368us) server_negotiation.cc:299] Negotiation successful
0806 10:57:18.010664 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 10:57:18.011710 15319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:57:18.103178 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 15193
I20260806 10:57:18.121645 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 15330
Aug 06 10:57:18 dist-test-slave-49h3 krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.3@KRBTEST.COM
I20260806 10:57:18.138954 15478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:18.122440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57162 (local address 127.30.218.3:35047)
0806 10:57:18.122638 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:18.122642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:18.122662 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:57:18.122811 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:18.122815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:18.122897 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:18.122984 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:57:18.124409 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:18.125248 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:18.126864 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:18.127299 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:18.127408 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:18.132310 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:57:18.132331 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:57:18.132348 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:57:18.132446 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:57:18.135576 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:18.136604 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:18.136612 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:18.136615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:18.136784 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:18.137244 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:18.137248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:18.137250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:18.137744 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:57:18.137998 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:18.138471 (+ 473us) server_negotiation.cc:299] Negotiation successful
0806 10:57:18.138683 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 10:57:18.139633 15456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:57:18.231179 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 15330
I20260806 10:57:18.248363 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 14957
I20260806 10:57:18.249667 15020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:57:18.362954 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 14957
I20260806 10:57:38.385648 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:46571
--webserver_interface=127.30.218.62
--webserver_port=43097
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:40935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:46571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:57:38.780470 15513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:57:38.780850 15513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:57:38.781001 15513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:57:38.788810 15513 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 10:57:38.788931 15513 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 10:57:38.788995 15513 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 10:57:38.789069 15513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:57:38.789146 15513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:57:38.789249 15513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:57:38.789314 15513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:57:38.789394 15513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:57:38.800176 15513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:40935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:46571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:46571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=43097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:57:38.801959 15513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:57:38.803659 15513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:57:38.833741 15520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:57:38.833781 15522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:57:38.833766 15519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:57:38.834720 15513 server_base.cc:1034] running on GCE node
I20260806 10:57:38.835729 15513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:57:38.837579 15513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:57:38.838809 15513 hybrid_clock.cc:648] HybridClock initialized: now 1786013858838735 us; error 46 us; skew 500 ppm
I20260806 10:57:38.839247 15513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:57:38 dist-test-slave-49h3 krb5kdc[14944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013858, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:57:38.843205 15513 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:57:38.845031 15513 webserver.cc:460] Webserver started at http://127.30.218.62:43097/ using document root <none> and password file <none>
I20260806 10:57:38.845952 15513 fs_manager.cc:359] Metadata directory not provided
I20260806 10:57:38.846091 15513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:57:38.852680 15513 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260806 10:57:38.855944 15529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:57:38.857414 15513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:57:38.857618 15513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "1ecd9ddddd834083b02461abaaf1fef0"
format_stamp: "Formatted at 2026-08-06 10:57:15 on dist-test-slave-49h3"
server_key: "4879fbb8790a0c277cb72cfeb48a5a99"
server_key_iv: "c66071c2053ca53c73563e975acf081e"
server_key_version: "encryptionkey@0"
I20260806 10:57:38.858758 15513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:57:38.881019 15513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:57:38.886375 15513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:57:38.886673 15513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:57:38.907795 15513 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:46571
I20260806 10:57:38.907815 15582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:46571 every 8 connection(s)
I20260806 10:57:38.909765 15513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:57:38.915120 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 15513
I20260806 10:57:38.916127 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:33447
--local_ip_for_outbound_sockets=127.30.218.1
--tserver_master_addrs=127.30.218.62:46571
--webserver_port=44377
--webserver_interface=127.30.218.1
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:40935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:57:38.918253 15583 sys_catalog.cc:263] Verifying existing consensus state
I20260806 10:57:38.922031 15583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0: Bootstrap starting.
I20260806 10:57:38.945910 15583 log.cc:824] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0: Log is configured to *not* fsync() on all Append() calls
I20260806 10:57:38.961364 15583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 10:57:38.962060 15583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0: Bootstrap complete.
I20260806 10:57:38.972860 15583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecd9ddddd834083b02461abaaf1fef0" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46571 } }
I20260806 10:57:38.973948 15583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ecd9ddddd834083b02461abaaf1fef0, State: Initialized, Role: FOLLOWER
I20260806 10:57:38.974745 15583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecd9ddddd834083b02461abaaf1fef0" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46571 } }
I20260806 10:57:38.975024 15583 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:57:38.975150 15583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:57:38.975327 15583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 10:57:38.979923 15583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecd9ddddd834083b02461abaaf1fef0" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46571 } }
I20260806 10:57:38.980532 15583 leader_election.cc:302] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ecd9ddddd834083b02461abaaf1fef0; no voters:
I20260806 10:57:38.981207 15583 leader_election.cc:288] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 10:57:38.981432 15587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 10:57:38.982432 15587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 2 LEADER]: Becoming Leader. State: Replica: 1ecd9ddddd834083b02461abaaf1fef0, State: Running, Role: LEADER
I20260806 10:57:38.983278 15587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecd9ddddd834083b02461abaaf1fef0" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46571 } }
I20260806 10:57:38.983563 15583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:57:38.986702 15589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ecd9ddddd834083b02461abaaf1fef0. Latest consensus state: current_term: 2 leader_uuid: "1ecd9ddddd834083b02461abaaf1fef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecd9ddddd834083b02461abaaf1fef0" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46571 } } }
I20260806 10:57:38.987195 15589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [sys.catalog]: This master's current role is: LEADER
I20260806 10:57:38.987797 15588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1ecd9ddddd834083b02461abaaf1fef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecd9ddddd834083b02461abaaf1fef0" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46571 } } }
I20260806 10:57:38.988163 15588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [sys.catalog]: This master's current role is: LEADER
I20260806 10:57:38.991974 15594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:57:39.001487 15594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:57:39.003159 15594 catalog_manager.cc:1259] Loaded cluster ID: 5b98922264d74c4d9a577c8a7c7444b5
I20260806 10:57:39.003325 15594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:57:39.007893 15594 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:57:39.009684 15594 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0: Loaded TSK: 0
I20260806 10:57:39.028677 15606 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 10:57:39.046970 15594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0: Generated new TSK 1
I20260806 10:57:39.047394 15594 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 10:57:39.344928 15585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:57:39.345264 15585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:57:39.345392 15585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:57:39.353041 15585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:57:39.353158 15585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:57:39.353310 15585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:57:39.364044 15585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:40935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:33447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=44377
--tserver_master_addrs=127.30.218.62:46571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:57:39.365624 15585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:57:39.367228 15585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:57:39.384317 15613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:57:39.384317 15612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:57:39.384891 15615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:57:39.385890 15585 server_base.cc:1034] running on GCE node
I20260806 10:57:39.386719 15585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:57:39.387977 15585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:57:39.389259 15585 hybrid_clock.cc:648] HybridClock initialized: now 1786013859389173 us; error 86 us; skew 500 ppm
I20260806 10:57:39.389680 15585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:57:39 dist-test-slave-49h3 krb5kdc[14944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013859, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:57:39.397347 15585 init.cc:377] Logged in from keytab as kudu/127.30.218.1@KRBTEST.COM (short username kudu)
I20260806 10:57:39.399255 15585 webserver.cc:460] Webserver started at http://127.30.218.1:44377/ using document root <none> and password file <none>
I20260806 10:57:39.400122 15585 fs_manager.cc:359] Metadata directory not provided
I20260806 10:57:39.400274 15585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:57:39.406704 15585 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 10:57:39.409822 15622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:57:39.411504 15585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:57:39.411701 15585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "4405ef6882ce4212bda93cdaed80d15f"
format_stamp: "Formatted at 2026-08-06 10:57:16 on dist-test-slave-49h3"
server_key: "63d4a353fadf48ced7062d43554e30da"
server_key_iv: "a5593b2145c040cc727567c411cd0a40"
server_key_version: "encryptionkey@0"
I20260806 10:57:39.412896 15585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:57:39.435582 15585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:57:39.440838 15585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:57:39.441143 15585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:57:39.442356 15585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:57:39.443933 15585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:57:39.444046 15585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:57:39.444173 15585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:57:39.444255 15585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:57:39.477949 15585 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:33447
I20260806 10:57:39.478010 15739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:33447 every 8 connection(s)
I20260806 10:57:39.479744 15585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:57:39.488989 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 15585
I20260806 10:57:39.489786 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:43451
--local_ip_for_outbound_sockets=127.30.218.2
--tserver_master_addrs=127.30.218.62:46571
--webserver_port=33919
--webserver_interface=127.30.218.2
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:40935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 10:57:39 dist-test-slave-49h3 krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013859, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:57:39.508592 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:39.483180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:42131 (local address 127.30.218.62:46571)
0806 10:57:39.484028 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:39.484044 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:39.484614 (+ 570us) server_negotiation.cc:407] Connection header received
0806 10:57:39.487158 (+ 2544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:39.487174 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:39.487608 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:39.487946 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:57:39.489863 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:39.491843 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:39.493834 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:39.494836 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:39.495051 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:39.499201 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:57:39.499215 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:57:39.499237 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:57:39.499299 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:57:39.503266 (+ 3967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:39.504335 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:39.504345 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:39.504354 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:39.504618 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:39.505250 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:39.505254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:39.505256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:39.505899 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:57:39.506090 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:39.507114 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0806 10:57:39.507489 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":448,"threads_started":1}
I20260806 10:57:39.508742 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:39.483416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.1:42131)
0806 10:57:39.484390 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:39.484441 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 10:57:39.486532 (+ 2091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:57:39.488139 (+ 1607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:57:39.488147 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:57:39.488820 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:57:39.489540 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:39.489565 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:39.492133 (+ 2568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:39.492138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:39.493642 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:39.493659 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:39.495086 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:39.495093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:39.495299 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:39.496049 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:57:39.496083 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:57:39.498786 (+ 2703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:57:39.503539 (+ 4753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:57:39.503549 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:57:39.503566 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:57:39.504138 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:57:39.504790 (+ 652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:57:39.504797 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:57:39.504800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:57:39.505110 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:57:39.506068 (+ 958us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:57:39.506077 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:57:39.506551 (+ 474us) client_negotiation.cc:769] Sending connection context
0806 10:57:39.506902 (+ 351us) client_negotiation.cc:240] Negotiation successful
0806 10:57:39.507507 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":366,"threads_started":1}
I20260806 10:57:39.511926 15740 heartbeater.cc:347] Connected to a master server at 127.30.218.62:46571
I20260806 10:57:39.512280 15740 heartbeater.cc:464] Registering TS with master...
I20260806 10:57:39.513137 15740 heartbeater.cc:511] Master 127.30.218.62:46571 requested a full tablet report, sending...
I20260806 10:57:39.515854 15547 ts_manager.cc:194] Registered new tserver with Master: 4405ef6882ce4212bda93cdaed80d15f (127.30.218.1:33447)
I20260806 10:57:39.517669 15547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.1@KRBTEST.COM'} at 127.30.218.1:42131
W20260806 10:57:39.867631 15744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:57:39.867949 15744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:57:39.868091 15744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:57:39.875623 15744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:57:39.875754 15744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:57:39.875916 15744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:57:39.886368 15744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:40935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:43451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=33919
--tserver_master_addrs=127.30.218.62:46571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:57:39.887840 15744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:57:39.889249 15744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:57:39.903254 15749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:57:39.903246 15752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:57:39.903297 15750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:57:39.903736 15744 server_base.cc:1034] running on GCE node
I20260806 10:57:39.904701 15744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:57:39.905735 15744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:57:39.906945 15744 hybrid_clock.cc:648] HybridClock initialized: now 1786013859906857 us; error 67 us; skew 500 ppm
I20260806 10:57:39.907316 15744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:57:39 dist-test-slave-49h3 krb5kdc[14944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013859, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:57:39.911206 15744 init.cc:377] Logged in from keytab as kudu/127.30.218.2@KRBTEST.COM (short username kudu)
I20260806 10:57:39.913031 15744 webserver.cc:460] Webserver started at http://127.30.218.2:33919/ using document root <none> and password file <none>
I20260806 10:57:39.914003 15744 fs_manager.cc:359] Metadata directory not provided
I20260806 10:57:39.914184 15744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:57:39.921890 15744 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s
I20260806 10:57:39.925691 15759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:57:39.927546 15744 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 10:57:39.927784 15744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "028b0d0d65d2450e8a3725c99cc5ccbb"
format_stamp: "Formatted at 2026-08-06 10:57:16 on dist-test-slave-49h3"
server_key: "1996d0ace5792c489b29cb157e3f09e5"
server_key_iv: "0538d1575046b4c88b2ab5d61a152592"
server_key_version: "encryptionkey@0"
I20260806 10:57:39.928998 15744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:57:39.945698 15744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:57:39.950480 15744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:57:39.950762 15744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:57:39.951798 15744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:57:39.953227 15744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:57:39.953332 15744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:57:39.953423 15744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:57:39.953518 15744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260806 10:57:39.985215 15744 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:43451
I20260806 10:57:39.985320 15874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:43451 every 8 connection(s)
I20260806 10:57:39.986745 15744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:57:39.987975 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 15744
I20260806 10:57:39.988782 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:35047
--local_ip_for_outbound_sockets=127.30.218.3
--tserver_master_addrs=127.30.218.62:46571
--webserver_port=42639
--webserver_interface=127.30.218.3
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:40935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 10:57:40 dist-test-slave-49h3 krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013859, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:57:40.009052 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:39.989566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:53335 (local address 127.30.218.62:46571)
0806 10:57:39.989780 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:39.989787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:39.991133 (+ 1346us) server_negotiation.cc:407] Connection header received
0806 10:57:39.993474 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:39.993481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:39.993612 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:39.993733 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:57:39.995768 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:39.996495 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:39.998614 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:39.999019 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:39.999116 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:40.003232 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:57:40.003237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:57:40.003240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:57:40.003274 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:57:40.005812 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:40.006688 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:40.006692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:40.006694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:40.006819 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:40.007274 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:40.007277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:40.007279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:40.007557 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:57:40.007674 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:40.008584 (+ 910us) server_negotiation.cc:299] Negotiation successful
0806 10:57:40.008796 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 10:57:40.010421 15877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:39.990075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.2:53335)
0806 10:57:39.990937 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:39.990986 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 10:57:39.993154 (+ 2168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:57:39.993988 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:57:39.993999 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:57:39.994757 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:57:39.995572 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:39.995598 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:39.996666 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:39.996672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:39.998473 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:39.998487 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:39.999241 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:39.999248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:39.999523 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:40.000286 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:57:40.000313 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:57:40.002982 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:57:40.005957 (+ 2975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:57:40.005965 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:57:40.005983 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:57:40.006559 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:57:40.006923 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:57:40.006928 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:57:40.006931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:57:40.007173 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:57:40.007688 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:57:40.007695 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:57:40.008135 (+ 440us) client_negotiation.cc:769] Sending connection context
0806 10:57:40.008446 (+ 311us) client_negotiation.cc:240] Negotiation successful
0806 10:57:40.009002 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":307,"threads_started":1}
I20260806 10:57:40.011906 15875 heartbeater.cc:347] Connected to a master server at 127.30.218.62:46571
I20260806 10:57:40.012250 15875 heartbeater.cc:464] Registering TS with master...
I20260806 10:57:40.013108 15875 heartbeater.cc:511] Master 127.30.218.62:46571 requested a full tablet report, sending...
I20260806 10:57:40.014861 15547 ts_manager.cc:194] Registered new tserver with Master: 028b0d0d65d2450e8a3725c99cc5ccbb (127.30.218.2:43451)
I20260806 10:57:40.015726 15547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.2@KRBTEST.COM'} at 127.30.218.2:53335
W20260806 10:57:40.339425 15878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:57:40.339743 15878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:57:40.339869 15878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:57:40.347272 15878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:57:40.347421 15878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:57:40.347595 15878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:57:40.357947 15878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:40935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:35047
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=42639
--tserver_master_addrs=127.30.218.62:46571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:57:40.359330 15878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:57:40.360698 15878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:57:40.373936 15884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:57:40.374053 15885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:57:40.374003 15887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:57:40.374686 15878 server_base.cc:1034] running on GCE node
I20260806 10:57:40.375299 15878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:57:40.376279 15878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:57:40.377501 15878 hybrid_clock.cc:648] HybridClock initialized: now 1786013860377447 us; error 42 us; skew 500 ppm
I20260806 10:57:40.377828 15878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:57:40 dist-test-slave-49h3 krb5kdc[14944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013860, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:57:40.381019 15878 init.cc:377] Logged in from keytab as kudu/127.30.218.3@KRBTEST.COM (short username kudu)
I20260806 10:57:40.382536 15878 webserver.cc:460] Webserver started at http://127.30.218.3:42639/ using document root <none> and password file <none>
I20260806 10:57:40.383263 15878 fs_manager.cc:359] Metadata directory not provided
I20260806 10:57:40.383428 15878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:57:40.389262 15878 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260806 10:57:40.392215 15894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:57:40.393760 15878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:57:40.393940 15878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "e42dd462ddfe4d979ee3662dc9f29bae"
format_stamp: "Formatted at 2026-08-06 10:57:17 on dist-test-slave-49h3"
server_key: "b1f97602c1a91ee92dc89f7941771bda"
server_key_iv: "72132c406671b929f6570c1acd6941d0"
server_key_version: "encryptionkey@0"
I20260806 10:57:40.394994 15878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:57:40.421722 15878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:57:40.426306 15878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:57:40.426587 15878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:57:40.427735 15878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:57:40.429193 15878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:57:40.429284 15878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:57:40.429394 15878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:57:40.429481 15878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:57:40.461483 15878 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:35047
I20260806 10:57:40.461571 16010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:35047 every 8 connection(s)
I20260806 10:57:40.462965 15878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:57:40.465047 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 15878
Aug 06 10:57:40 dist-test-slave-49h3 krb5kdc[14944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013860, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:57:40.484661 16015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:40.466343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37112 (local address 127.30.218.62:46571)
0806 10:57:40.469821 (+ 3478us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:40.469825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:40.469840 (+ 15us) server_negotiation.cc:407] Connection header received
0806 10:57:40.469883 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:40.469887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:40.469937 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:40.470025 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:57:40.471872 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.472622 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.474342 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.474753 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.474863 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:40.477857 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:57:40.477862 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:57:40.477865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:57:40.477897 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:57:40.481727 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:40.482444 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:40.482450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:40.482454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:40.482619 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:40.483058 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:40.483064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:40.483067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:40.483398 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:57:40.483736 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:40.483987 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 10:57:40.484220 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":3385,"thread_start_us":3253,"threads_started":1}
I20260806 10:57:40.485023 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:40.465699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:58767 (local address 127.30.218.62:46571)
0806 10:57:40.465932 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:40.465938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:40.467127 (+ 1189us) server_negotiation.cc:407] Connection header received
0806 10:57:40.468953 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:40.468959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:40.469052 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:40.469141 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:57:40.470962 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.471722 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.473587 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.473984 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.474075 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:40.478243 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:57:40.478249 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:57:40.478253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:57:40.478303 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:57:40.481629 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:40.482557 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:40.482562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:40.482565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:40.482690 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:40.483135 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:40.483139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:40.483141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:40.483509 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:57:40.483653 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:40.484504 (+ 851us) server_negotiation.cc:299] Negotiation successful
0806 10:57:40.484736 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 10:57:40.486666 16014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:40.466155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.3:58767)
0806 10:57:40.466986 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:40.467022 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 10:57:40.468696 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:57:40.469393 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:57:40.469402 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:57:40.469979 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:57:40.470777 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:40.470804 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.471892 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.471897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:40.473428 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:40.473473 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.474199 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.474207 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:40.474459 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:40.475163 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:57:40.475194 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:57:40.477997 (+ 2803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:57:40.481804 (+ 3807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:57:40.481813 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:57:40.481827 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:57:40.482435 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:57:40.482808 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:57:40.482811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:57:40.482813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:57:40.482989 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:57:40.483647 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:57:40.483655 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:57:40.484025 (+ 370us) client_negotiation.cc:769] Sending connection context
0806 10:57:40.485191 (+ 1166us) client_negotiation.cc:240] Negotiation successful
0806 10:57:40.485659 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":341,"threads_started":1}
I20260806 10:57:40.487908 16011 heartbeater.cc:347] Connected to a master server at 127.30.218.62:46571
I20260806 10:57:40.488214 16011 heartbeater.cc:464] Registering TS with master...
I20260806 10:57:40.488942 16011 heartbeater.cc:511] Master 127.30.218.62:46571 requested a full tablet report, sending...
I20260806 10:57:40.490368 15547 ts_manager.cc:194] Registered new tserver with Master: e42dd462ddfe4d979ee3662dc9f29bae (127.30.218.3:35047)
I20260806 10:57:40.491168 15547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.3@KRBTEST.COM'} at 127.30.218.3:58767
I20260806 10:57:40.500099 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:57:40.507266 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:40.501171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37126 (local address 127.30.218.62:46571)
0806 10:57:40.501368 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:40.501375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:40.501790 (+ 415us) server_negotiation.cc:407] Connection header received
0806 10:57:40.502023 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:40.502027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:40.502184 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:40.502327 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:57:40.502838 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.503668 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.505160 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.505628 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.505736 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:40.506295 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:57:40.506606 (+ 311us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 10:57:40.506985 (+ 379us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37126: token expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260806 10:57:40.507390 15742 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37126: token expired
W20260806 10:57:40.508045 15468 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.30.218.62:46571)
I20260806 10:57:40.521642 15740 heartbeater.cc:503] Master 127.30.218.62:46571 was elected leader, sending a full tablet report...
I20260806 10:57:40.521970 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:40.508740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37136 (local address 127.30.218.62:46571)
0806 10:57:40.508926 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:40.508932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:40.508947 (+ 15us) server_negotiation.cc:407] Connection header received
0806 10:57:40.508986 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:40.508989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:40.509071 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:40.509161 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:57:40.510588 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.511373 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.512892 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.513302 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.513414 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:40.516347 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:57:40.516352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:57:40.516355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:57:40.516393 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:57:40.519107 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:40.519870 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:40.519875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:40.519877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:40.520004 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:40.520395 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:40.520399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:40.520401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:40.520777 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:57:40.521013 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:40.521353 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 10:57:40.521627 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 10:57:40.525818 15546 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:57:40.533318 15546 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:57:40.558365 16021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:40.545303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50286 (local address 127.30.218.3:35047)
0806 10:57:40.546146 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:40.546155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:40.547179 (+ 1024us) server_negotiation.cc:407] Connection header received
0806 10:57:40.547289 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:40.547298 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:40.547581 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:40.547769 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:57:40.548891 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.550749 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.553931 (+ 3182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.555665 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.555816 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:40.556120 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:40.557771 (+ 1651us) server_negotiation.cc:299] Negotiation successful
0806 10:57:40.558002 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":483,"threads_started":1}
I20260806 10:57:40.559258 16022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:40.545101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:35047 (local address 127.0.0.1:50286)
0806 10:57:40.546760 (+ 1659us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:40.546798 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 10:57:40.546976 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:57:40.548115 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:57:40.548124 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:57:40.548166 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:57:40.548635 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:40.548648 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.550957 (+ 2309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.550964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:40.553675 (+ 2711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:40.553699 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.556329 (+ 2630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.556337 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:40.556620 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:40.556686 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 10:57:40.556850 (+ 164us) client_negotiation.cc:240] Negotiation successful
0806 10:57:40.557109 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1379,"spinlock_wait_cycles":2304,"thread_start_us":1163,"threads_started":1}
I20260806 10:57:40.563773 15943 tablet_service.cc:1467] Processing CreateTablet for tablet fc8753b965624bb1a7e6f6526f4f8f2e (DEFAULT_TABLE table=test-table [id=d1b3b89f38394261a4c33a4a06c96e71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:57:40.565408 15943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc8753b965624bb1a7e6f6526f4f8f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:57:40.568346 16024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:40.545496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:33447 (local address 127.0.0.1:54102)
0806 10:57:40.546759 (+ 1263us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:40.546798 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 10:57:40.546976 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:57:40.560229 (+ 13253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:57:40.560235 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:57:40.560264 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:57:40.560672 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:40.560688 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.563104 (+ 2416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.563110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:40.565192 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:40.565209 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.567453 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.567459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:40.567646 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:40.567684 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 10:57:40.567787 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 10:57:40.567929 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1064,"spinlock_wait_cycles":2304,"thread_start_us":1798,"threads_started":1}
I20260806 10:57:40.568370 16023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:40.545829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54102 (local address 127.30.218.1:33447)
0806 10:57:40.559267 (+ 13438us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:40.559277 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:40.559309 (+ 32us) server_negotiation.cc:407] Connection header received
0806 10:57:40.559424 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:40.559433 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:40.559739 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:40.559956 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:57:40.560878 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.562852 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.565377 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.567184 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.567327 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:40.567560 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:40.567850 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 10:57:40.568038 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13258,"thread_start_us":13085,"threads_started":1}
I20260806 10:57:40.572724 16025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:40.547654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:43451 (local address 127.0.0.1:34322)
0806 10:57:40.563131 (+ 15477us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:40.563148 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:57:40.563292 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:57:40.564349 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:57:40.564353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:57:40.564374 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:57:40.564694 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:40.564704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.566502 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.566507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:40.569335 (+ 2828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:40.569351 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.571587 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.571601 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:40.572055 (+ 454us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:40.572097 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 10:57:40.572194 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 10:57:40.572355 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15338,"thread_start_us":15209,"threads_started":1}
I20260806 10:57:40.572359 15673 tablet_service.cc:1467] Processing CreateTablet for tablet fc8753b965624bb1a7e6f6526f4f8f2e (DEFAULT_TABLE table=test-table [id=d1b3b89f38394261a4c33a4a06c96e71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:57:40.573577 16026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:40.547895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34322 (local address 127.30.218.2:43451)
0806 10:57:40.562639 (+ 14744us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:40.562650 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:40.563647 (+ 997us) server_negotiation.cc:407] Connection header received
0806 10:57:40.563746 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:40.563752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:40.563937 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:40.564136 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:57:40.564850 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.566338 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.569547 (+ 3209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.571016 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.571172 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:40.571364 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:40.573004 (+ 1640us) server_negotiation.cc:299] Negotiation successful
0806 10:57:40.573221 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14533,"thread_start_us":15491,"threads_started":1}
I20260806 10:57:40.574172 15673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc8753b965624bb1a7e6f6526f4f8f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:57:40.577883 15808 tablet_service.cc:1467] Processing CreateTablet for tablet fc8753b965624bb1a7e6f6526f4f8f2e (DEFAULT_TABLE table=test-table [id=d1b3b89f38394261a4c33a4a06c96e71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:57:40.579497 15808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc8753b965624bb1a7e6f6526f4f8f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:57:40.581542 16027 tablet_bootstrap.cc:492] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae: Bootstrap starting.
I20260806 10:57:40.584995 16027 tablet_bootstrap.cc:654] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae: Neither blocks nor log segments found. Creating new log.
I20260806 10:57:40.586163 16027 log.cc:824] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae: Log is configured to *not* fsync() on all Append() calls
I20260806 10:57:40.591066 16029 tablet_bootstrap.cc:492] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f: Bootstrap starting.
I20260806 10:57:40.593056 16027 tablet_bootstrap.cc:492] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae: No bootstrap required, opened a new log
I20260806 10:57:40.593348 16027 ts_tablet_manager.cc:1397] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260806 10:57:40.594857 16029 tablet_bootstrap.cc:654] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f: Neither blocks nor log segments found. Creating new log.
I20260806 10:57:40.595379 16030 tablet_bootstrap.cc:492] T fc8753b965624bb1a7e6f6526f4f8f2e P 028b0d0d65d2450e8a3725c99cc5ccbb: Bootstrap starting.
I20260806 10:57:40.596174 16029 log.cc:824] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f: Log is configured to *not* fsync() on all Append() calls
I20260806 10:57:40.598737 16030 tablet_bootstrap.cc:654] T fc8753b965624bb1a7e6f6526f4f8f2e P 028b0d0d65d2450e8a3725c99cc5ccbb: Neither blocks nor log segments found. Creating new log.
I20260806 10:57:40.599536 16029 tablet_bootstrap.cc:492] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f: No bootstrap required, opened a new log
I20260806 10:57:40.599812 16029 ts_tablet_manager.cc:1397] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260806 10:57:40.599860 16030 log.cc:824] T fc8753b965624bb1a7e6f6526f4f8f2e P 028b0d0d65d2450e8a3725c99cc5ccbb: Log is configured to *not* fsync() on all Append() calls
I20260806 10:57:40.603652 16030 tablet_bootstrap.cc:492] T fc8753b965624bb1a7e6f6526f4f8f2e P 028b0d0d65d2450e8a3725c99cc5ccbb: No bootstrap required, opened a new log
I20260806 10:57:40.603560 16027 raft_consensus.cc:348] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35047 } } peers { permanent_uuid: "4405ef6882ce4212bda93cdaed80d15f" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33447 } } peers { permanent_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 43451 } }
I20260806 10:57:40.603907 16030 ts_tablet_manager.cc:1397] T fc8753b965624bb1a7e6f6526f4f8f2e P 028b0d0d65d2450e8a3725c99cc5ccbb: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260806 10:57:40.604035 16027 raft_consensus.cc:374] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:57:40.604137 16027 raft_consensus.cc:729] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e42dd462ddfe4d979ee3662dc9f29bae, State: Initialized, Role: FOLLOWER
I20260806 10:57:40.604813 16027 consensus_queue.cc:260] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35047 } } peers { permanent_uuid: "4405ef6882ce4212bda93cdaed80d15f" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33447 } } peers { permanent_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 43451 } }
I20260806 10:57:40.606065 16011 heartbeater.cc:503] Master 127.30.218.62:46571 was elected leader, sending a full tablet report...
I20260806 10:57:40.606788 16027 ts_tablet_manager.cc:1428] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae: Time spent starting tablet: real 0.013s user 0.000s sys 0.014s
I20260806 10:57:40.610050 16029 raft_consensus.cc:348] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35047 } } peers { permanent_uuid: "4405ef6882ce4212bda93cdaed80d15f" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33447 } } peers { permanent_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 43451 } }
I20260806 10:57:40.610541 16029 raft_consensus.cc:374] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:57:40.610648 16029 raft_consensus.cc:729] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4405ef6882ce4212bda93cdaed80d15f, State: Initialized, Role: FOLLOWER
I20260806 10:57:40.611254 16029 consensus_queue.cc:260] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35047 } } peers { permanent_uuid: "4405ef6882ce4212bda93cdaed80d15f" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33447 } } peers { permanent_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 43451 } }
I20260806 10:57:40.613188 16029 ts_tablet_manager.cc:1428] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260806 10:57:40.614045 16030 raft_consensus.cc:348] T fc8753b965624bb1a7e6f6526f4f8f2e P 028b0d0d65d2450e8a3725c99cc5ccbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35047 } } peers { permanent_uuid: "4405ef6882ce4212bda93cdaed80d15f" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33447 } } peers { permanent_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 43451 } }
I20260806 10:57:40.614524 16030 raft_consensus.cc:374] T fc8753b965624bb1a7e6f6526f4f8f2e P 028b0d0d65d2450e8a3725c99cc5ccbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:57:40.614642 16030 raft_consensus.cc:729] T fc8753b965624bb1a7e6f6526f4f8f2e P 028b0d0d65d2450e8a3725c99cc5ccbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 028b0d0d65d2450e8a3725c99cc5ccbb, State: Initialized, Role: FOLLOWER
I20260806 10:57:40.615278 16030 consensus_queue.cc:260] T fc8753b965624bb1a7e6f6526f4f8f2e P 028b0d0d65d2450e8a3725c99cc5ccbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35047 } } peers { permanent_uuid: "4405ef6882ce4212bda93cdaed80d15f" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33447 } } peers { permanent_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 43451 } }
I20260806 10:57:40.616366 15875 heartbeater.cc:503] Master 127.30.218.62:46571 was elected leader, sending a full tablet report...
I20260806 10:57:40.617162 16030 ts_tablet_manager.cc:1428] T fc8753b965624bb1a7e6f6526f4f8f2e P 028b0d0d65d2450e8a3725c99cc5ccbb: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
I20260806 10:57:40.707406 16034 raft_consensus.cc:482] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:57:40.707724 16034 raft_consensus.cc:504] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35047 } } peers { permanent_uuid: "4405ef6882ce4212bda93cdaed80d15f" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33447 } } peers { permanent_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 43451 } }
I20260806 10:57:40.709530 16034 leader_election.cc:288] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4405ef6882ce4212bda93cdaed80d15f (127.30.218.1:33447), 028b0d0d65d2450e8a3725c99cc5ccbb (127.30.218.2:43451)
W20260806 10:57:40.714608 16012 tablet.cc:2369] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:57:40.716132 16023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:40.709955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:34901 (local address 127.30.218.1:33447)
0806 10:57:40.710184 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:40.710190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:40.710238 (+ 48us) server_negotiation.cc:407] Connection header received
0806 10:57:40.710427 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:40.710433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:40.710516 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:40.710630 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:57:40.711117 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.712052 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.714181 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.715284 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.715462 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:40.715541 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:40.715823 (+ 282us) server_negotiation.cc:299] Negotiation successful
0806 10:57:40.715928 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 10:57:40.716254 16014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:40.709933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:33447 (local address 127.30.218.3:34901)
0806 10:57:40.710145 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:40.710167 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:57:40.710274 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:57:40.710697 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:57:40.710702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:57:40.710735 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:57:40.710958 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:40.710970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.712253 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.712257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:40.714019 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:40.714036 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.715458 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.715464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:40.715659 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:40.715697 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 10:57:40.715804 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 10:57:40.715956 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 10:57:40.717067 16038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:40.710444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:43451 (local address 127.30.218.3:55259)
0806 10:57:40.711223 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:40.711238 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 10:57:40.711357 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:57:40.711661 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:57:40.711664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:57:40.711681 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:57:40.711950 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:40.711958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.712995 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.713001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:40.714839 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:57:40.714854 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.716287 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.716293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:57:40.716461 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:40.716487 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 10:57:40.716685 (+ 198us) client_negotiation.cc:240] Negotiation successful
0806 10:57:40.716823 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":471,"threads_started":1}
I20260806 10:57:40.717095 16026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:40.710592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:55259 (local address 127.30.218.2:43451)
0806 10:57:40.710809 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:40.710816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:40.711339 (+ 523us) server_negotiation.cc:407] Connection header received
0806 10:57:40.711473 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:40.711478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:40.711558 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:40.711650 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:57:40.712094 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.712831 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.714990 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.716051 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.716203 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:40.716278 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:40.716707 (+ 429us) server_negotiation.cc:299] Negotiation successful
0806 10:57:40.716836 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 10:57:40.717427 15694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc8753b965624bb1a7e6f6526f4f8f2e" candidate_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4405ef6882ce4212bda93cdaed80d15f" is_pre_election: true
I20260806 10:57:40.718092 15694 raft_consensus.cc:2454] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e42dd462ddfe4d979ee3662dc9f29bae in term 0.
I20260806 10:57:40.718190 15829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc8753b965624bb1a7e6f6526f4f8f2e" candidate_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb" is_pre_election: true
I20260806 10:57:40.718700 15829 raft_consensus.cc:2454] T fc8753b965624bb1a7e6f6526f4f8f2e P 028b0d0d65d2450e8a3725c99cc5ccbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e42dd462ddfe4d979ee3662dc9f29bae in term 0.
I20260806 10:57:40.718852 15896 leader_election.cc:302] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4405ef6882ce4212bda93cdaed80d15f, e42dd462ddfe4d979ee3662dc9f29bae; no voters:
I20260806 10:57:40.719302 16034 raft_consensus.cc:2781] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:57:40.719453 16034 raft_consensus.cc:482] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:57:40.719550 16034 raft_consensus.cc:3037] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:57:40.721872 16034 raft_consensus.cc:504] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35047 } } peers { permanent_uuid: "4405ef6882ce4212bda93cdaed80d15f" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33447 } } peers { permanent_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 43451 } }
I20260806 10:57:40.722544 16034 leader_election.cc:288] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae [CANDIDATE]: Term 1 election: Requested vote from peers 4405ef6882ce4212bda93cdaed80d15f (127.30.218.1:33447), 028b0d0d65d2450e8a3725c99cc5ccbb (127.30.218.2:43451)
I20260806 10:57:40.723176 15694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc8753b965624bb1a7e6f6526f4f8f2e" candidate_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4405ef6882ce4212bda93cdaed80d15f"
I20260806 10:57:40.723229 15829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc8753b965624bb1a7e6f6526f4f8f2e" candidate_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb"
I20260806 10:57:40.723433 15694 raft_consensus.cc:3037] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:57:40.723433 15829 raft_consensus.cc:3037] T fc8753b965624bb1a7e6f6526f4f8f2e P 028b0d0d65d2450e8a3725c99cc5ccbb [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:57:40.726953 15694 raft_consensus.cc:2454] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e42dd462ddfe4d979ee3662dc9f29bae in term 1.
I20260806 10:57:40.726953 15829 raft_consensus.cc:2454] T fc8753b965624bb1a7e6f6526f4f8f2e P 028b0d0d65d2450e8a3725c99cc5ccbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e42dd462ddfe4d979ee3662dc9f29bae in term 1.
I20260806 10:57:40.727504 15896 leader_election.cc:302] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 028b0d0d65d2450e8a3725c99cc5ccbb, e42dd462ddfe4d979ee3662dc9f29bae; no voters:
I20260806 10:57:40.727849 16034 raft_consensus.cc:2781] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:57:40.728364 16034 raft_consensus.cc:686] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae [term 1 LEADER]: Becoming Leader. State: Replica: e42dd462ddfe4d979ee3662dc9f29bae, State: Running, Role: LEADER
I20260806 10:57:40.728955 16034 consensus_queue.cc:237] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35047 } } peers { permanent_uuid: "4405ef6882ce4212bda93cdaed80d15f" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33447 } } peers { permanent_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 43451 } }
W20260806 10:57:40.732442 15741 tablet.cc:2369] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:57:40.735162 15546 catalog_manager.cc:5495] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae reported cstate change: term changed from 0 to 1, leader changed from <none> to e42dd462ddfe4d979ee3662dc9f29bae (127.30.218.3). New cstate: current_term: 1 leader_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42dd462ddfe4d979ee3662dc9f29bae" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 35047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4405ef6882ce4212bda93cdaed80d15f" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 43451 } health_report { overall_health: UNKNOWN } } }
W20260806 10:57:40.738785 15876 tablet.cc:2369] T fc8753b965624bb1a7e6f6526f4f8f2e P 028b0d0d65d2450e8a3725c99cc5ccbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:57:40.755686 16021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:40.750302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50300 (local address 127.30.218.3:35047)
0806 10:57:40.750528 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:40.750534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:40.750554 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:57:40.750705 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:40.750709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:40.750806 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:40.750946 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:57:40.751301 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.752067 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.753495 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.753919 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.754018 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:40.754513 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:57:40.755106 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:57:40.755233 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:40.755315 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 10:57:40.755397 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 10:57:40.765002 15829 raft_consensus.cc:1261] T fc8753b965624bb1a7e6f6526f4f8f2e P 028b0d0d65d2450e8a3725c99cc5ccbb [term 1 FOLLOWER]: Refusing update from remote peer e42dd462ddfe4d979ee3662dc9f29bae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:57:40.765005 15694 raft_consensus.cc:1261] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f [term 1 FOLLOWER]: Refusing update from remote peer e42dd462ddfe4d979ee3662dc9f29bae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:57:40.766160 16034 consensus_queue.cc:1035] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae [LEADER]: Connected to new peer: Peer: permanent_uuid: "028b0d0d65d2450e8a3725c99cc5ccbb" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 43451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:57:40.766559 16039 consensus_queue.cc:1035] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae [LEADER]: Connected to new peer: Peer: permanent_uuid: "4405ef6882ce4212bda93cdaed80d15f" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 33447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:57:40.779223 16043 mvcc.cc:205] Tried to move back new op lower bound from 7315512773682245632 to 7315512773550485504. Current Snapshot: MvccSnapshot[applied={T|T < 7315512773682245632}]
I20260806 10:57:40.780342 16042 mvcc.cc:205] Tried to move back new op lower bound from 7315512773682245632 to 7315512773550485504. Current Snapshot: MvccSnapshot[applied={T|T < 7315512773682245632}]
I20260806 10:57:40.783934 16044 mvcc.cc:205] Tried to move back new op lower bound from 7315512773682245632 to 7315512773550485504. Current Snapshot: MvccSnapshot[applied={T|T < 7315512773682245632}]
I20260806 10:57:40.798262 15547 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:57:40.798614 15547 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:57:40.802997 15547 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0: Sending DeleteTablet for 3 replicas of tablet fc8753b965624bb1a7e6f6526f4f8f2e
I20260806 10:57:40.804394 15943 tablet_service.cc:1514] Processing DeleteTablet for tablet fc8753b965624bb1a7e6f6526f4f8f2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:57:40 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:50286
I20260806 10:57:40.804445 15808 tablet_service.cc:1514] Processing DeleteTablet for tablet fc8753b965624bb1a7e6f6526f4f8f2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:57:40 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:34322
I20260806 10:57:40.804571 15673 tablet_service.cc:1514] Processing DeleteTablet for tablet fc8753b965624bb1a7e6f6526f4f8f2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:57:40 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:54102
I20260806 10:57:40.806063 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 15585
I20260806 10:57:40.815433 16052 tablet_replica.cc:330] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae: stopping tablet replica
I20260806 10:57:40.816030 16052 raft_consensus.cc:2229] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:57:40.816920 16052 raft_consensus.cc:2258] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:57:40.817936 16054 tablet_replica.cc:330] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f: stopping tablet replica
I20260806 10:57:40.818447 16054 raft_consensus.cc:2229] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:57:40.819051 16054 raft_consensus.cc:2258] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:57:40.821890 16054 ts_tablet_manager.cc:1905] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:57:40.826092 16053 tablet_replica.cc:330] T fc8753b965624bb1a7e6f6526f4f8f2e P 028b0d0d65d2450e8a3725c99cc5ccbb: stopping tablet replica
I20260806 10:57:40.826632 16053 raft_consensus.cc:2229] T fc8753b965624bb1a7e6f6526f4f8f2e P 028b0d0d65d2450e8a3725c99cc5ccbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:57:40.827232 16053 raft_consensus.cc:2258] T fc8753b965624bb1a7e6f6526f4f8f2e P 028b0d0d65d2450e8a3725c99cc5ccbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:57:40.830097 16052 ts_tablet_manager.cc:1905] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:57:40.832196 16023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:40.807939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54108 (local address 127.30.218.1:33447)
0806 10:57:40.808149 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:40.808166 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:40.808188 (+ 22us) server_negotiation.cc:407] Connection header received
0806 10:57:40.808260 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:40.808266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:40.808383 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:40.808491 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:57:40.810899 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.812287 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.814692 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.815455 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.815592 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:40.821129 (+ 5537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:57:40.821161 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:57:40.821182 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:57:40.821248 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:57:40.826954 (+ 5706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:40.827929 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:40.827940 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:40.827943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:40.828156 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:40.828726 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:40.828731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:40.828735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:40.829280 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:57:40.830885 (+ 1605us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:40.831073 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 10:57:40.831772 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 10:57:40.833606 15735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:57:40.834317 16053 ts_tablet_manager.cc:1905] T fc8753b965624bb1a7e6f6526f4f8f2e P 028b0d0d65d2450e8a3725c99cc5ccbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:57:40.839890 16052 ts_tablet_manager.cc:1918] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:57:40.840147 16052 log.cc:1196] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/fc8753b965624bb1a7e6f6526f4f8f2e
I20260806 10:57:40.840817 16052 ts_tablet_manager.cc:1939] T fc8753b965624bb1a7e6f6526f4f8f2e P e42dd462ddfe4d979ee3662dc9f29bae: Deleting consensus metadata
I20260806 10:57:40.841661 16053 ts_tablet_manager.cc:1918] T fc8753b965624bb1a7e6f6526f4f8f2e P 028b0d0d65d2450e8a3725c99cc5ccbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:57:40.841852 16053 log.cc:1196] T fc8753b965624bb1a7e6f6526f4f8f2e P 028b0d0d65d2450e8a3725c99cc5ccbb: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/fc8753b965624bb1a7e6f6526f4f8f2e
I20260806 10:57:40.832265 16054 ts_tablet_manager.cc:1918] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:57:40.842272 16054 log.cc:1196] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/wal/wals/fc8753b965624bb1a7e6f6526f4f8f2e
I20260806 10:57:40.842794 16053 ts_tablet_manager.cc:1939] T fc8753b965624bb1a7e6f6526f4f8f2e P 028b0d0d65d2450e8a3725c99cc5ccbb: Deleting consensus metadata
I20260806 10:57:40.843223 15532 catalog_manager.cc:4841] TS e42dd462ddfe4d979ee3662dc9f29bae (127.30.218.3:35047): tablet fc8753b965624bb1a7e6f6526f4f8f2e (table test-table [id=d1b3b89f38394261a4c33a4a06c96e71]) successfully deleted
I20260806 10:57:40.844897 15531 catalog_manager.cc:4841] TS 028b0d0d65d2450e8a3725c99cc5ccbb (127.30.218.2:43451): tablet fc8753b965624bb1a7e6f6526f4f8f2e (table test-table [id=d1b3b89f38394261a4c33a4a06c96e71]) successfully deleted
I20260806 10:57:40.945748 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 15585
I20260806 10:57:40.945745 16054 ts_tablet_manager.cc:1939] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f: Deleting consensus metadata
I20260806 10:57:40.964520 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 15744
W20260806 10:57:40.964602 15531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.218.1:33447 (error 108)
I20260806 10:57:40.966879 16025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:40.966357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:33447 (local address 127.0.0.1:54110)
0806 10:57:40.966556 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:40.966697 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:33447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 10:57:40.967173 15531 catalog_manager.cc:4584] TS 4405ef6882ce4212bda93cdaed80d15f (127.30.218.1:33447): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc8753b965624bb1a7e6f6526f4f8f2e: Network error: Client connection negotiation failed: client connection to 127.30.218.1:33447: connect: Connection refused (error 111)
I20260806 10:57:40.981364 16026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:40.965563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34324 (local address 127.30.218.2:43451)
0806 10:57:40.965865 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:40.965871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:40.965894 (+ 23us) server_negotiation.cc:407] Connection header received
0806 10:57:40.966034 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:40.966041 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:40.966129 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:40.966244 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:57:40.967748 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.968897 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.970377 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:40.971067 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:40.971205 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:40.973956 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:57:40.973978 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:57:40.974001 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:57:40.974051 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:57:40.978099 (+ 4048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:40.978938 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:40.978949 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:40.978954 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:40.979121 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:40.979510 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:40.979515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:40.979519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:40.980044 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:57:40.980367 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:40.980733 (+ 366us) server_negotiation.cc:299] Negotiation successful
0806 10:57:40.980985 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260806 10:57:40.982090 15870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:57:41.021090 16025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:41.020447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:33447 (local address 127.0.0.1:54116)
0806 10:57:41.020749 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:41.020889 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:33447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260806 10:57:41.022495 16025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:41.021995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:33447 (local address 127.0.0.1:54128)
0806 10:57:41.022161 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:41.022263 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:33447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 10:57:41.074478 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 15744
I20260806 10:57:41.083494 16025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:41.082912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:33447 (local address 127.0.0.1:54144)
0806 10:57:41.083178 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:41.083299 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:33447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260806 10:57:41.084843 16025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:41.084407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:33447 (local address 127.0.0.1:54160)
0806 10:57:41.084598 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:41.084693 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:33447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 10:57:41.092346 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 15878
I20260806 10:57:41.109942 16021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:41.093171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50306 (local address 127.30.218.3:35047)
0806 10:57:41.093384 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 10:57:41.093391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:57:41.093417 (+ 26us) server_negotiation.cc:407] Connection header received
0806 10:57:41.093696 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:57:41.093701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:57:41.093767 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:57:41.093851 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:57:41.095772 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:41.096777 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:41.098922 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:57:41.099488 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:57:41.099622 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:57:41.103158 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:57:41.103175 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:57:41.103193 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:57:41.103251 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:57:41.106719 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:41.107729 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:41.107738 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:41.107740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:41.107896 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:57:41.108341 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:57:41.108344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:57:41.108350 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:57:41.108753 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:57:41.109007 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0806 10:57:41.109368 (+ 361us) server_negotiation.cc:299] Negotiation successful
0806 10:57:41.109611 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 10:57:41.110747 16006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:57:41.166420 16025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:41.165773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:33447 (local address 127.0.0.1:54172)
0806 10:57:41.166043 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:41.166184 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:33447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260806 10:57:41.167579 16025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:41.167121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:33447 (local address 127.0.0.1:54174)
0806 10:57:41.167346 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:41.167440 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:33447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260806 10:57:41.237895 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 15878
I20260806 10:57:41.257536 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 15513
I20260806 10:57:41.258706 15578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:57:41.377092 16025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:57:41.376333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:33447 (local address 127.0.0.1:54186)
0806 10:57:41.376755 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0806 10:57:41.376877 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:33447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220}
I20260806 10:57:41.377147 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 15513
I20260806 10:58:01.400005 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:46571
--webserver_interface=127.30.218.62
--webserver_port=43097
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:40935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:46571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:58:01.769701 16060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:01.770079 16060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:01.770241 16060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:01.777953 16060 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 10:58:01.778100 16060 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 10:58:01.778172 16060 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 10:58:01.778242 16060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:58:01.778321 16060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:01.778419 16060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:01.778482 16060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:58:01.778558 16060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:58:01.789254 16060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:40935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:46571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:46571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=43097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:01.790764 16060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:01.792236 16060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:01.805958 16065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:01.805898 16066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:01.806228 16068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:01.806986 16060 server_base.cc:1034] running on GCE node
I20260806 10:58:01.808140 16060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:01.810225 16060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:01.811530 16060 hybrid_clock.cc:648] HybridClock initialized: now 1786013881811398 us; error 71 us; skew 500 ppm
I20260806 10:58:01.812072 16060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:58:01 dist-test-slave-49h3 krb5kdc[14944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013881, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:58:01.820889 16060 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:58:01.822860 16060 webserver.cc:460] Webserver started at http://127.30.218.62:43097/ using document root <none> and password file <none>
I20260806 10:58:01.823783 16060 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:01.823928 16060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:01.831346 16060 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 10:58:01.834923 16075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:01.836692 16060 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 10:58:01.836875 16060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "1ecd9ddddd834083b02461abaaf1fef0"
format_stamp: "Formatted at 2026-08-06 10:57:15 on dist-test-slave-49h3"
server_key: "4879fbb8790a0c277cb72cfeb48a5a99"
server_key_iv: "c66071c2053ca53c73563e975acf081e"
server_key_version: "encryptionkey@0"
I20260806 10:58:01.838116 16060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:01.888525 16060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:01.893420 16060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:01.893728 16060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:01.914654 16060 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:46571
I20260806 10:58:01.914709 16127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:46571 every 8 connection(s)
I20260806 10:58:01.916380 16060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:58:01.919008 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 16060
I20260806 10:58:01.919950 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:33447
--local_ip_for_outbound_sockets=127.30.218.1
--tserver_master_addrs=127.30.218.62:46571
--webserver_port=44377
--webserver_interface=127.30.218.1
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:40935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:58:01.924335 16128 sys_catalog.cc:263] Verifying existing consensus state
I20260806 10:58:01.928058 16128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0: Bootstrap starting.
I20260806 10:58:01.954838 16128 log.cc:824] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:01.981684 16128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 10:58:01.982641 16128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0: Bootstrap complete.
I20260806 10:58:01.998198 16128 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecd9ddddd834083b02461abaaf1fef0" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46571 } }
I20260806 10:58:01.999784 16128 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ecd9ddddd834083b02461abaaf1fef0, State: Initialized, Role: FOLLOWER
I20260806 10:58:02.000679 16128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecd9ddddd834083b02461abaaf1fef0" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46571 } }
I20260806 10:58:02.001044 16128 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:58:02.001192 16128 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:58:02.001435 16128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 10:58:02.008054 16128 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecd9ddddd834083b02461abaaf1fef0" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46571 } }
I20260806 10:58:02.008824 16128 leader_election.cc:302] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ecd9ddddd834083b02461abaaf1fef0; no voters:
I20260806 10:58:02.009745 16128 leader_election.cc:288] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 10:58:02.009887 16132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 10:58:02.011152 16132 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [term 3 LEADER]: Becoming Leader. State: Replica: 1ecd9ddddd834083b02461abaaf1fef0, State: Running, Role: LEADER
I20260806 10:58:02.011706 16132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecd9ddddd834083b02461abaaf1fef0" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46571 } }
I20260806 10:58:02.012590 16128 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:58:02.014164 16134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ecd9ddddd834083b02461abaaf1fef0. Latest consensus state: current_term: 3 leader_uuid: "1ecd9ddddd834083b02461abaaf1fef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecd9ddddd834083b02461abaaf1fef0" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46571 } } }
I20260806 10:58:02.014560 16134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:02.015141 16133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1ecd9ddddd834083b02461abaaf1fef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecd9ddddd834083b02461abaaf1fef0" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46571 } } }
I20260806 10:58:02.015460 16133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0 [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:02.017249 16137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:58:02.027369 16137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:58:02.028939 16137 catalog_manager.cc:1259] Loaded cluster ID: 5b98922264d74c4d9a577c8a7c7444b5
I20260806 10:58:02.029053 16137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:58:02.033293 16137 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:58:02.034981 16137 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0: Loaded TSK: 0
I20260806 10:58:02.035123 16137 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1ecd9ddddd834083b02461abaaf1fef0: Loaded TSK: 1
F20260806 10:58:02.036288 16137 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f29071809ad google::LogMessage::Fail() at ??:0
@ 0x7f290718388a google::LogMessage::SendToLog() at ??:0
@ 0x7f29071804de google::LogMessage::Flush() at ??:0
@ 0x7f2907181a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f290aee47d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f290aed373e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f291cbaf1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f291cbae783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f291cc3a2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f291cc3a0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f291cc845e9 std::function<>::operator()() at ??:0
@ 0x7f291cbb2cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f291cbb0c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f291cc38610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f291cc38439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2909dcc345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2909dcf91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f2909dcf799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2909daaeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f290d0846db start_thread at ??:0
@ 0x7f290601a61f clone at ??:0
W20260806 10:58:02.414942 16130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:02.415290 16130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:02.415405 16130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:02.423097 16130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:02.423230 16130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:02.423362 16130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:58:02.433817 16130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:40935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:33447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=44377
--tserver_master_addrs=127.30.218.62:46571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:02.435408 16130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:02.436962 16130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:02.451490 16176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:02.451536 16179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:02.451584 16177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:02.452421 16130 server_base.cc:1034] running on GCE node
I20260806 10:58:02.453043 16130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:02.454087 16130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:02.455320 16130 hybrid_clock.cc:648] HybridClock initialized: now 1786013882455251 us; error 33 us; skew 500 ppm
I20260806 10:58:02.455653 16130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:58:02 dist-test-slave-49h3 krb5kdc[14944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013882, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:58:02.459165 16130 init.cc:377] Logged in from keytab as kudu/127.30.218.1@KRBTEST.COM (short username kudu)
I20260806 10:58:02.460793 16130 webserver.cc:460] Webserver started at http://127.30.218.1:44377/ using document root <none> and password file <none>
I20260806 10:58:02.461583 16130 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:02.461714 16130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:02.467526 16130 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 10:58:02.470719 16186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:02.472198 16130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:58:02.472347 16130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "4405ef6882ce4212bda93cdaed80d15f"
format_stamp: "Formatted at 2026-08-06 10:57:16 on dist-test-slave-49h3"
server_key: "63d4a353fadf48ced7062d43554e30da"
server_key_iv: "a5593b2145c040cc727567c411cd0a40"
server_key_version: "encryptionkey@0"
I20260806 10:58:02.473517 16130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:02.494738 16130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:02.499253 16130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:02.499540 16130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:02.500770 16130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:58:02.503057 16193 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260806 10:58:02.509742 16193 ts_tablet_manager.cc:1852] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260806 10:58:02.511065 16193 ts_tablet_manager.cc:1905] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:58:02.520989 16193 ts_tablet_manager.cc:1918] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:58:02.521430 16193 ts_tablet_manager.cc:1939] T fc8753b965624bb1a7e6f6526f4f8f2e P 4405ef6882ce4212bda93cdaed80d15f: Deleting consensus metadata
I20260806 10:58:02.521976 16130 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260806 10:58:02.522116 16130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.020s user 0.001s sys 0.000s
I20260806 10:58:02.522269 16130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:58:02.522357 16130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:02.560986 16130 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:33447
I20260806 10:58:02.561022 16307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:33447 every 8 connection(s)
I20260806 10:58:02.562678 16130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:58:02.568500 16311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:02.566269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.1:44665)
0806 10:58:02.567127 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:02.567422 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":330,"threads_started":1}
W20260806 10:58:02.569193 16308 heartbeater.cc:644] Failed to heartbeat to 127.30.218.62:46571 (0 consecutive failures): Network error: Failed to ping master at 127.30.218.62:46571: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
I20260806 10:58:02.570076 16311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:02.569747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.1:45943)
0806 10:58:02.569887 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:02.569961 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260806 10:58:02.570930 16311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:02.570556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.1:37709)
0806 10:58:02.570675 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:02.570764 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 10:58:02.571172 16308 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 10:58:02.571468 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 16130
I20260806 10:58:02.572259 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:43451
--local_ip_for_outbound_sockets=127.30.218.2
--tserver_master_addrs=127.30.218.62:46571
--webserver_port=33919
--webserver_interface=127.30.218.2
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:40935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:58:02.971777 16312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:02.972187 16312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:02.972332 16312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:02.980720 16312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:02.980906 16312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:02.981084 16312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:58:02.992739 16312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:40935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:43451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=33919
--tserver_master_addrs=127.30.218.62:46571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:02.994634 16312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:02.996383 16312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:03.013087 16318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:03.013213 16317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:03.013154 16320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:03.013952 16312 server_base.cc:1034] running on GCE node
I20260806 10:58:03.014729 16312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:03.015846 16312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:03.017082 16312 hybrid_clock.cc:648] HybridClock initialized: now 1786013883017011 us; error 58 us; skew 500 ppm
I20260806 10:58:03.017511 16312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:58:03 dist-test-slave-49h3 krb5kdc[14944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013883, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:58:03.021122 16312 init.cc:377] Logged in from keytab as kudu/127.30.218.2@KRBTEST.COM (short username kudu)
I20260806 10:58:03.022977 16312 webserver.cc:460] Webserver started at http://127.30.218.2:33919/ using document root <none> and password file <none>
I20260806 10:58:03.023844 16312 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:03.023977 16312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:03.030615 16312 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 10:58:03.033919 16327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:03.035573 16312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:58:03.035771 16312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "028b0d0d65d2450e8a3725c99cc5ccbb"
format_stamp: "Formatted at 2026-08-06 10:57:16 on dist-test-slave-49h3"
server_key: "1996d0ace5792c489b29cb157e3f09e5"
server_key_iv: "0538d1575046b4c88b2ab5d61a152592"
server_key_version: "encryptionkey@0"
I20260806 10:58:03.036947 16312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:03.074697 16312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:03.079563 16312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:03.079854 16312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:03.080988 16312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:58:03.082665 16312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:58:03.082785 16312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:03.082903 16312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:58:03.082974 16312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:03.116016 16312 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:43451
I20260806 10:58:03.116123 16443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:43451 every 8 connection(s)
I20260806 10:58:03.117769 16312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:58:03.123216 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 16312
I20260806 10:58:03.123677 16446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:03.121176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.2:59717)
0806 10:58:03.122056 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:03.122349 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":332,"threads_started":1}
I20260806 10:58:03.124109 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:35047
--local_ip_for_outbound_sockets=127.30.218.3
--tserver_master_addrs=127.30.218.62:46571
--webserver_port=42639
--webserver_interface=127.30.218.3
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:40935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:58:03.124511 16444 heartbeater.cc:644] Failed to heartbeat to 127.30.218.62:46571 (0 consecutive failures): Network error: Failed to ping master at 127.30.218.62:46571: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
I20260806 10:58:03.125566 16446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:03.125099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.2:43583)
0806 10:58:03.125280 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:03.125378 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 10:58:03.126703 16446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:03.126329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.2:35357)
0806 10:58:03.126480 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:03.126565 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 10:58:03.126963 16444 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 10:58:03.511785 16447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:03.512144 16447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:03.512249 16447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:03.520078 16447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:03.520223 16447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:03.520346 16447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:58:03.531272 16447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:40935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:35047
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=42639
--tserver_master_addrs=127.30.218.62:46571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:03.532749 16447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:03.534461 16447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:03.549260 16456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:03.549187 16453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:03.549237 16454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:03.550287 16447 server_base.cc:1034] running on GCE node
I20260806 10:58:03.550958 16447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:03.552078 16447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:03.553318 16447 hybrid_clock.cc:648] HybridClock initialized: now 1786013883553247 us; error 60 us; skew 500 ppm
I20260806 10:58:03.553707 16447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:58:03 dist-test-slave-49h3 krb5kdc[14944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013883, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:58:03.557190 16447 init.cc:377] Logged in from keytab as kudu/127.30.218.3@KRBTEST.COM (short username kudu)
I20260806 10:58:03.558945 16447 webserver.cc:460] Webserver started at http://127.30.218.3:42639/ using document root <none> and password file <none>
I20260806 10:58:03.559752 16447 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:03.559896 16447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:03.567723 16447 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:58:03.570844 16463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:03.572882 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:03.571945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.1:35273)
0806 10:58:03.572507 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:03.572613 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":291,"threads_started":1}
I20260806 10:58:03.572942 16447 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260806 10:58:03.573160 16447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "e42dd462ddfe4d979ee3662dc9f29bae"
format_stamp: "Formatted at 2026-08-06 10:57:17 on dist-test-slave-49h3"
server_key: "b1f97602c1a91ee92dc89f7941771bda"
server_key_iv: "72132c406671b929f6570c1acd6941d0"
server_key_version: "encryptionkey@0"
I20260806 10:58:03.574347 16447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:03.595878 16447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:03.600709 16447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:03.600993 16447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:03.602147 16447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:58:03.603708 16447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:58:03.603825 16447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:03.603945 16447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:58:03.604020 16447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:03.636694 16447 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:35047
I20260806 10:58:03.636756 16577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:35047 every 8 connection(s)
I20260806 10:58:03.638243 16447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:58:03.641356 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 16447
I20260806 10:58:03.643517 16580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:03.641157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.3:54901)
0806 10:58:03.642035 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:03.642361 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":340,"threads_started":1}
W20260806 10:58:03.644222 16578 heartbeater.cc:644] Failed to heartbeat to 127.30.218.62:46571 (0 consecutive failures): Network error: Failed to ping master at 127.30.218.62:46571: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
I20260806 10:58:03.645164 16580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:03.644719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.3:43321)
0806 10:58:03.644924 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:03.645009 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 10:58:03.646090 16580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:03.645754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.3:44071)
0806 10:58:03.645868 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:03.645946 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260806 10:58:03.646320 16578 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 10:58:04.128793 16582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:04.127747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.2:51711)
0806 10:58:04.128365 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:04.128489 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":315,"threads_started":1}
I20260806 10:58:04.574761 16583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:04.573898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.1:47021)
0806 10:58:04.574442 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:04.574543 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":290,"threads_started":1}
I20260806 10:58:04.647948 16584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:04.647015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.3:54453)
0806 10:58:04.647551 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:04.647680 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":253,"threads_started":1}
I20260806 10:58:05.130885 16585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:05.130017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.2:45783)
0806 10:58:05.130564 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:05.130660 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":295,"threads_started":1}
I20260806 10:58:05.576824 16586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:05.575868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.1:60947)
0806 10:58:05.576456 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:05.576592 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":289,"threads_started":1}
I20260806 10:58:05.650203 16587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:05.649168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.3:41639)
0806 10:58:05.649873 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:05.649977 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":422,"threads_started":1}
I20260806 10:58:06.133009 16588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:06.132080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.2:34307)
0806 10:58:06.132680 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:06.132784 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":324,"threads_started":1}
I20260806 10:58:06.579054 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:06.578062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.1:45223)
0806 10:58:06.578693 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:06.578797 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":360,"threads_started":1}
I20260806 10:58:06.652256 16590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:06.651296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.3:39095)
0806 10:58:06.651885 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:06.652000 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":300,"threads_started":1}
I20260806 10:58:07.135537 16591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:07.134455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.2:41685)
0806 10:58:07.135145 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:07.135286 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":341,"threads_started":1}
I20260806 10:58:07.581208 16592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:07.580242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.1:34937)
0806 10:58:07.580871 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:07.581002 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":287,"threads_started":1}
I20260806 10:58:07.654537 16593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:07.653511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.3:36447)
0806 10:58:07.654168 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:07.654295 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":314,"threads_started":1}
I20260806 10:58:08.137768 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:08.136836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.2:40591)
0806 10:58:08.137405 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:08.137545 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":303,"threads_started":1}
I20260806 10:58:08.583176 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:08.582289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.1:35317)
0806 10:58:08.582841 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:08.582939 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":290,"threads_started":1}
I20260806 10:58:08.656723 16597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:08.655762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.3:60225)
0806 10:58:08.656380 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:08.656498 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":323,"threads_started":1}
I20260806 10:58:09.139812 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:09.138936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.2:35965)
0806 10:58:09.139486 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:09.139593 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":281,"threads_started":1}
I20260806 10:58:09.585484 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:09.584296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.1:37765)
0806 10:58:09.585009 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:09.585129 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":338,"threads_started":1}
I20260806 10:58:09.658842 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:09.657906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.3:36709)
0806 10:58:09.658435 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:09.658583 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":276,"threads_started":1}
I20260806 10:58:10.142108 16601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:10.140992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.2:34073)
0806 10:58:10.141710 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:10.141844 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":376,"threads_started":1}
I20260806 10:58:10.587419 16602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:10.586530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.1:38065)
0806 10:58:10.587093 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:10.587191 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":310,"threads_started":1}
I20260806 10:58:10.660744 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:10.659814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.3:59691)
0806 10:58:10.660384 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:10.660485 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":241,"threads_started":1}
I20260806 10:58:11.143959 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:11.143122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.2:49979)
0806 10:58:11.143650 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:11.143751 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":293,"threads_started":1}
I20260806 10:58:11.589570 16605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:11.588586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.1:47389)
0806 10:58:11.589183 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:11.589310 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":275,"threads_started":1}
I20260806 10:58:11.662746 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:11.661878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.3:51163)
0806 10:58:11.662433 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:11.662540 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":279,"threads_started":1}
I20260806 10:58:12.145953 16607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:12.145030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.2:37323)
0806 10:58:12.145620 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:12.145750 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":260,"threads_started":1}
I20260806 10:58:12.591821 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:12.590761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.1:60721)
0806 10:58:12.591395 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:12.591536 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":298,"threads_started":1}
I20260806 10:58:12.664646 16609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:12.663755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.3:57815)
0806 10:58:12.664328 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:12.664426 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":292,"threads_started":1}
I20260806 10:58:13.147956 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:13.147040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.2:33303)
0806 10:58:13.147625 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:13.147730 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":338,"threads_started":1}
I20260806 10:58:13.594012 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:13.592983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.1:46409)
0806 10:58:13.593648 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:13.593773 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":357,"threads_started":1}
I20260806 10:58:13.666652 16612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:13.665764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.3:60935)
0806 10:58:13.666313 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:13.666418 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":304,"threads_started":1}
I20260806 10:58:14.149899 16613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:14.148966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.2:46213)
0806 10:58:14.149545 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:14.149656 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":279,"threads_started":1}
I20260806 10:58:14.595916 16614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:14.595044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.1:41853)
0806 10:58:14.595569 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:14.595676 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":285,"threads_started":1}
I20260806 10:58:14.668705 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:14.667799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.3:57129)
0806 10:58:14.668354 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:14.668469 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":270,"threads_started":1}
I20260806 10:58:15.151868 16617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:15.150989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.2:34907)
0806 10:58:15.151542 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:15.151648 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":287,"threads_started":1}
I20260806 10:58:15.597991 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:15.596942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.1:56965)
0806 10:58:15.597616 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:15.597750 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":340,"threads_started":1}
I20260806 10:58:15.670765 16619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:15.669881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.3:44923)
0806 10:58:15.670447 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:15.670559 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":281,"threads_started":1}
I20260806 10:58:16.153810 16620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:16.152912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.2:53959)
0806 10:58:16.153421 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:16.153571 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":261,"threads_started":1}
I20260806 10:58:16.600039 16621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:16.599126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.1:47241)
0806 10:58:16.599692 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:16.599809 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":261,"threads_started":1}
I20260806 10:58:16.672837 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:16.671897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.3:48245)
0806 10:58:16.672477 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:16.672612 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":287,"threads_started":1}
I20260806 10:58:17.155766 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:17.154928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.2:40479)
0806 10:58:17.155437 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:17.155552 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":262,"threads_started":1}
I20260806 10:58:17.602355 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:17.601431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.1:58187)
0806 10:58:17.602006 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:17.602134 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":275,"threads_started":1}
I20260806 10:58:17.674739 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:17.673847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.3:43837)
0806 10:58:17.674375 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:17.674485 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":255,"threads_started":1}
I20260806 10:58:18.158025 16626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:18.157076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.2:50359)
0806 10:58:18.157634 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:18.157742 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":287,"threads_started":1}
I20260806 10:58:18.604391 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:18.603432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.1:58783)
0806 10:58:18.603999 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:18.604126 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":305,"threads_started":1}
I20260806 10:58:18.648962 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 16130
I20260806 10:58:18.669543 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:18.650186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50208 (local address 127.30.218.1:33447)
0806 10:58:18.650884 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:18.650902 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:18.650959 (+ 57us) server_negotiation.cc:407] Connection header received
0806 10:58:18.653296 (+ 2337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:18.653310 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:18.653984 (+ 674us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:18.654341 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:18.655885 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:18.657239 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:18.658675 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:18.659471 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:18.659614 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:18.662860 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:18.662870 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:18.662887 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:18.662947 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:18.666070 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:18.666855 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:18.666862 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:18.666870 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:18.667056 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:18.667468 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:18.667471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:18.667473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:18.668078 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:18.668326 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:18.668763 (+ 437us) server_negotiation.cc:299] Negotiation successful
0806 10:58:18.669168 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":316,"threads_started":1}
I20260806 10:58:18.671146 16303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:18.676898 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:18.676040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:46571 (local address 127.30.218.3:42861)
0806 10:58:18.676590 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:18.676698 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.62:46571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":273,"threads_started":1}
I20260806 10:58:18.764565 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 16130
I20260806 10:58:18.786751 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 16312
I20260806 10:58:18.807047 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:18.787982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37690 (local address 127.30.218.2:43451)
0806 10:58:18.788643 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:18.788662 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:18.788709 (+ 47us) server_negotiation.cc:407] Connection header received
0806 10:58:18.790610 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:18.790623 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:18.791147 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:18.791504 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:18.792933 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:18.794326 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:18.795819 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:18.796714 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:18.796875 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:18.799927 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:18.799936 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:18.799955 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:18.800001 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:18.803253 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:18.804052 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:18.804060 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:18.804068 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:18.804261 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:18.804647 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:18.804651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:18.804653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:18.805264 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:18.805633 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:18.806129 (+ 496us) server_negotiation.cc:299] Negotiation successful
0806 10:58:18.806624 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":308,"threads_started":1}
I20260806 10:58:18.808709 16439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:18.900159 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 16312
I20260806 10:58:18.921659 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 16447
I20260806 10:58:18.943169 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:18.923013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59300 (local address 127.30.218.3:35047)
0806 10:58:18.923807 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:18.923827 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:18.923884 (+ 57us) server_negotiation.cc:407] Connection header received
0806 10:58:18.926118 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:18.926132 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:18.926724 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:18.927106 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:18.928683 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:18.930209 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:18.931496 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:18.932236 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:18.932413 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:18.935575 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:18.935585 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:18.935606 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:18.935666 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:18.939323 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:18.940146 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:18.940154 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:18.940163 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:18.940402 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:18.940756 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:18.940759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:18.940761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:18.941563 (+ 802us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:18.941858 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:18.942252 (+ 394us) server_negotiation.cc:299] Negotiation successful
0806 10:58:18.942764 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":410,"threads_started":1}
I20260806 10:58:18.944994 16573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:19.034722 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 16447
2026-08-06T10:58:19Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65095 ms)
[----------] 30 tests from SecurityITest (309441 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:58:19 dist-test-slave-49h3 krb5kdc[16637](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:58:19 dist-test-slave-49h3 krb5kdc[16637](info): set up 2 sockets
Aug 06 10:58:19 dist-test-slave-49h3 krb5kdc[16637](info): commencing operation
krb5kdc: starting...
W20260806 10:58:20.135452 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.061s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:58:20 dist-test-slave-49h3 krb5kdc[16637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:58:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:58:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:58:20.468151 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:45655
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:45655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:58:20.822808 16651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:20.823171 16651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:20.823319 16651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:20.830925 16651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:58:20.831077 16651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:20.831166 16651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:20.831251 16651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:58:20.831313 16651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:58:20.841517 16651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:45655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:45655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:20.842908 16651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:20.844264 16651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:20.856309 16658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:20.856346 16660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:20.856390 16657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:20.856911 16651 server_base.cc:1034] running on GCE node
I20260806 10:58:20.857748 16651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:20.859157 16651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:20.860378 16651 hybrid_clock.cc:648] HybridClock initialized: now 1786013900860310 us; error 48 us; skew 500 ppm
I20260806 10:58:20.860721 16651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:58:20 dist-test-slave-49h3 krb5kdc[16637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013900, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:58:20.868184 16651 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:58:20.869990 16651 webserver.cc:460] Webserver started at http://127.30.218.62:45583/ using document root <none> and password file <none>
I20260806 10:58:20.870772 16651 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:20.870903 16651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:20.871222 16651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:20.874636 16651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "9f2335aa7d6240e092d45672e3cd1bd0"
format_stamp: "Formatted at 2026-08-06 10:58:20 on dist-test-slave-49h3"
server_key: "44687d347b9101dafd7d33301af625e9"
server_key_iv: "50e6dd2bd79306466324ba5394c1941b"
server_key_version: "encryptionkey@0"
I20260806 10:58:20.875483 16651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "9f2335aa7d6240e092d45672e3cd1bd0"
format_stamp: "Formatted at 2026-08-06 10:58:20 on dist-test-slave-49h3"
server_key: "44687d347b9101dafd7d33301af625e9"
server_key_iv: "50e6dd2bd79306466324ba5394c1941b"
server_key_version: "encryptionkey@0"
I20260806 10:58:20.881479 16651 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260806 10:58:20.886116 16667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:20.887804 16651 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 10:58:20.887996 16651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "9f2335aa7d6240e092d45672e3cd1bd0"
format_stamp: "Formatted at 2026-08-06 10:58:20 on dist-test-slave-49h3"
server_key: "44687d347b9101dafd7d33301af625e9"
server_key_iv: "50e6dd2bd79306466324ba5394c1941b"
server_key_version: "encryptionkey@0"
I20260806 10:58:20.888206 16651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:20.920940 16651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:20.925262 16651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:20.925590 16651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:20.944775 16651 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:45655
I20260806 10:58:20.944802 16720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:45655 every 8 connection(s)
I20260806 10:58:20.946214 16651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:58:20.950127 16721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:20.951959 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 16651
I20260806 10:58:20.952190 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:58:20.953001 31592 external_mini_cluster.cc:1428] Setting key 6e42571e51bb2bf0d757191a30dc0fc3
I20260806 10:58:20.963301 16721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f2335aa7d6240e092d45672e3cd1bd0: Bootstrap starting.
I20260806 10:58:20.967578 16721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f2335aa7d6240e092d45672e3cd1bd0: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:20.968962 16721 log.cc:824] T 00000000000000000000000000000000 P 9f2335aa7d6240e092d45672e3cd1bd0: Log is configured to *not* fsync() on all Append() calls
Aug 06 10:58:20 dist-test-slave-49h3 krb5kdc[16637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:58:20.972831 16721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f2335aa7d6240e092d45672e3cd1bd0: No bootstrap required, opened a new log
I20260806 10:58:20.980048 16724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:20.955171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41412 (local address 127.30.218.62:45655)
0806 10:58:20.956035 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:20.956047 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:20.956091 (+ 44us) server_negotiation.cc:407] Connection header received
0806 10:58:20.957932 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:20.957954 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:20.958370 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:20.958705 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:20.960708 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:20.962157 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:20.964123 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:20.964880 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:20.965052 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:20.970345 (+ 5293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:20.970354 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:20.970371 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:20.970416 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:20.974453 (+ 4037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:20.975550 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:20.975561 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:20.975572 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:20.975845 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:20.976239 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:20.976243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:20.976247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:20.977073 (+ 826us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:20.977399 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:20.977886 (+ 487us) server_negotiation.cc:299] Negotiation successful
0806 10:58:20.978432 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":401,"threads_started":1}
I20260806 10:58:20.983464 16721 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f2335aa7d6240e092d45672e3cd1bd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2335aa7d6240e092d45672e3cd1bd0" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 45655 } }
I20260806 10:58:20.983906 16721 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f2335aa7d6240e092d45672e3cd1bd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:20.983987 16721 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f2335aa7d6240e092d45672e3cd1bd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f2335aa7d6240e092d45672e3cd1bd0, State: Initialized, Role: FOLLOWER
I20260806 10:58:20.984576 16721 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f2335aa7d6240e092d45672e3cd1bd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2335aa7d6240e092d45672e3cd1bd0" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 45655 } }
I20260806 10:58:20.984825 16721 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f2335aa7d6240e092d45672e3cd1bd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:58:20.984972 16721 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f2335aa7d6240e092d45672e3cd1bd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:58:20.985121 16721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f2335aa7d6240e092d45672e3cd1bd0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:20.987653 16721 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f2335aa7d6240e092d45672e3cd1bd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2335aa7d6240e092d45672e3cd1bd0" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 45655 } }
I20260806 10:58:20.988196 16721 leader_election.cc:302] T 00000000000000000000000000000000 P 9f2335aa7d6240e092d45672e3cd1bd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f2335aa7d6240e092d45672e3cd1bd0; no voters:
I20260806 10:58:20.988827 16721 leader_election.cc:288] T 00000000000000000000000000000000 P 9f2335aa7d6240e092d45672e3cd1bd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:58:20.988983 16726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f2335aa7d6240e092d45672e3cd1bd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:58:20.989698 16726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f2335aa7d6240e092d45672e3cd1bd0 [term 1 LEADER]: Becoming Leader. State: Replica: 9f2335aa7d6240e092d45672e3cd1bd0, State: Running, Role: LEADER
I20260806 10:58:20.990324 16726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f2335aa7d6240e092d45672e3cd1bd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2335aa7d6240e092d45672e3cd1bd0" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 45655 } }
I20260806 10:58:20.990975 16721 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f2335aa7d6240e092d45672e3cd1bd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:58:20.993152 16727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f2335aa7d6240e092d45672e3cd1bd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f2335aa7d6240e092d45672e3cd1bd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2335aa7d6240e092d45672e3cd1bd0" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 45655 } } }
I20260806 10:58:20.993089 16728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f2335aa7d6240e092d45672e3cd1bd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f2335aa7d6240e092d45672e3cd1bd0. Latest consensus state: current_term: 1 leader_uuid: "9f2335aa7d6240e092d45672e3cd1bd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2335aa7d6240e092d45672e3cd1bd0" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 45655 } } }
I20260806 10:58:20.993702 16728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f2335aa7d6240e092d45672e3cd1bd0 [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:20.993587 16727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f2335aa7d6240e092d45672e3cd1bd0 [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:20.994678 16731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:58:21.000873 16731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:58:21.011230 16731 catalog_manager.cc:1347] Generated new cluster ID: 9328e17732b54973b04b5eef286f6a0f
I20260806 10:58:21.011355 16731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:58:21.026551 16731 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:58:21.027912 16731 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:58:21.035449 16731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f2335aa7d6240e092d45672e3cd1bd0: Generated new TSK 0
I20260806 10:58:21.036305 16731 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
I20260806 10:58:21.122617 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:45655
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:58:21.477686 16748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:21.477999 16748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:21.478152 16748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:21.486202 16748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:21.486340 16748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:21.486495 16748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:58:21.497262 16748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:45655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:21.498651 16748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:21.499910 16748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:21.512462 16754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:21.512573 16756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:21.513118 16753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:21.513551 16748 server_base.cc:1034] running on GCE node
I20260806 10:58:21.514233 16748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:21.515242 16748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:21.516471 16748 hybrid_clock.cc:648] HybridClock initialized: now 1786013901516404 us; error 57 us; skew 500 ppm
I20260806 10:58:21.516819 16748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:58:21 dist-test-slave-49h3 krb5kdc[16637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013901, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:58:21.520134 16748 init.cc:377] Logged in from keytab as kudu/127.30.218.1@KRBTEST.COM (short username kudu)
I20260806 10:58:21.521672 16748 webserver.cc:460] Webserver started at http://127.30.218.1:34333/ using document root <none> and password file <none>
I20260806 10:58:21.522363 16748 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:21.522476 16748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:21.522838 16748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:21.526139 16748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "c23a95014736428f8329c83ebdec5ceb"
format_stamp: "Formatted at 2026-08-06 10:58:21 on dist-test-slave-49h3"
server_key: "246649697693ae3a69ef17f3fc9c1862"
server_key_iv: "c4f2618c012e34b6711e0e842df763ae"
server_key_version: "encryptionkey@0"
I20260806 10:58:21.526919 16748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "c23a95014736428f8329c83ebdec5ceb"
format_stamp: "Formatted at 2026-08-06 10:58:21 on dist-test-slave-49h3"
server_key: "246649697693ae3a69ef17f3fc9c1862"
server_key_iv: "c4f2618c012e34b6711e0e842df763ae"
server_key_version: "encryptionkey@0"
I20260806 10:58:21.532110 16748 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 10:58:21.535657 16764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:21.537036 16748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:58:21.537214 16748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "c23a95014736428f8329c83ebdec5ceb"
format_stamp: "Formatted at 2026-08-06 10:58:21 on dist-test-slave-49h3"
server_key: "246649697693ae3a69ef17f3fc9c1862"
server_key_iv: "c4f2618c012e34b6711e0e842df763ae"
server_key_version: "encryptionkey@0"
I20260806 10:58:21.537424 16748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:21.564842 16748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:21.569517 16748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:21.569835 16748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:21.570870 16748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:58:21.572364 16748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:58:21.572453 16748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:21.572588 16748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:58:21.572642 16748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:21.606494 16748 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:38647
I20260806 10:58:21.606546 16880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:38647 every 8 connection(s)
I20260806 10:58:21.607878 16748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:58:21.616652 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 16748
I20260806 10:58:21.616850 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:58:21.617771 31592 external_mini_cluster.cc:1428] Setting key 0e4c63435cb9841043c53dd9d6b63248
Aug 06 10:58:21 dist-test-slave-49h3 krb5kdc[16637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013901, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:58:21.630180 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:21.610655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:51133 (local address 127.30.218.62:45655)
0806 10:58:21.611128 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:21.611133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:21.612047 (+ 914us) server_negotiation.cc:407] Connection header received
0806 10:58:21.614108 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:21.614114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:21.614207 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:21.614311 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:21.616087 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:21.616864 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:21.619233 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:21.619702 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:21.619826 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:21.623895 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:21.623901 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:21.623904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:21.623945 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:21.626618 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:21.627512 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:21.627517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:21.627519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:21.627673 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:21.628160 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:21.628164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:21.628166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:21.628472 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:21.628663 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:21.629510 (+ 847us) server_negotiation.cc:299] Negotiation successful
0806 10:58:21.629794 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":242,"threads_started":1}
I20260806 10:58:21.631273 16884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:21.611098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:45655 (local address 127.30.218.1:51133)
0806 10:58:21.611862 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:21.611915 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:21.613785 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:21.614600 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:21.614610 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:21.615211 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:58:21.615865 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:21.615887 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:21.617126 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:21.617134 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:21.619029 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:21.619054 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:21.619973 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:21.619982 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:21.620247 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:21.620985 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:58:21.621010 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:58:21.623610 (+ 2600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:58:21.626808 (+ 3198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:58:21.626818 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:58:21.626835 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:58:21.627358 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:58:21.627797 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:58:21.627802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:58:21.627806 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:58:21.628028 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:58:21.628619 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:58:21.628628 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:58:21.629046 (+ 418us) client_negotiation.cc:769] Sending connection context
0806 10:58:21.629324 (+ 278us) client_negotiation.cc:240] Negotiation successful
0806 10:58:21.629987 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":284,"threads_started":1}
I20260806 10:58:21.633059 16881 heartbeater.cc:347] Connected to a master server at 127.30.218.62:45655
I20260806 10:58:21.633469 16881 heartbeater.cc:464] Registering TS with master...
I20260806 10:58:21.634433 16881 heartbeater.cc:511] Master 127.30.218.62:45655 requested a full tablet report, sending...
I20260806 10:58:21.637123 16685 ts_manager.cc:194] Registered new tserver with Master: c23a95014736428f8329c83ebdec5ceb (127.30.218.1:38647)
I20260806 10:58:21.639246 16685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.1@KRBTEST.COM'} at 127.30.218.1:51133
WARNING: no policy specified for kudu/127.30.218.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
I20260806 10:58:21.690366 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:45655
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:58:22.039474 16888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:22.039778 16888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:22.039896 16888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:22.047345 16888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:22.047462 16888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:22.047623 16888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:58:22.057387 16888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:45655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:22.058665 16888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:22.059926 16888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:22.072816 16896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:22.072897 16893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:22.072851 16894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:22.073563 16888 server_base.cc:1034] running on GCE node
I20260806 10:58:22.074183 16888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:22.075165 16888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:22.076368 16888 hybrid_clock.cc:648] HybridClock initialized: now 1786013902076321 us; error 34 us; skew 500 ppm
I20260806 10:58:22.076679 16888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:58:22 dist-test-slave-49h3 krb5kdc[16637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013902, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:58:22.079710 16888 init.cc:377] Logged in from keytab as kudu/127.30.218.2@KRBTEST.COM (short username kudu)
I20260806 10:58:22.081189 16888 webserver.cc:460] Webserver started at http://127.30.218.2:35277/ using document root <none> and password file <none>
I20260806 10:58:22.081897 16888 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:22.082019 16888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:22.082319 16888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:22.085122 16888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "4cd782cd07cd4b7193abaf9a8208e214"
format_stamp: "Formatted at 2026-08-06 10:58:22 on dist-test-slave-49h3"
server_key: "b4d8c72b36ded9ddc0aa37e24a4f3556"
server_key_iv: "0500982f0dd1ce473051167bbaab72a1"
server_key_version: "encryptionkey@0"
I20260806 10:58:22.085916 16888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "4cd782cd07cd4b7193abaf9a8208e214"
format_stamp: "Formatted at 2026-08-06 10:58:22 on dist-test-slave-49h3"
server_key: "b4d8c72b36ded9ddc0aa37e24a4f3556"
server_key_iv: "0500982f0dd1ce473051167bbaab72a1"
server_key_version: "encryptionkey@0"
I20260806 10:58:22.091018 16888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 10:58:22.094856 16903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:22.096371 16888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:58:22.096554 16888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "4cd782cd07cd4b7193abaf9a8208e214"
format_stamp: "Formatted at 2026-08-06 10:58:22 on dist-test-slave-49h3"
server_key: "b4d8c72b36ded9ddc0aa37e24a4f3556"
server_key_iv: "0500982f0dd1ce473051167bbaab72a1"
server_key_version: "encryptionkey@0"
I20260806 10:58:22.096776 16888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:22.130358 16888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:22.134548 16888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:22.134837 16888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:22.135785 16888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:58:22.137090 16888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:58:22.137167 16888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:22.137307 16888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:58:22.137363 16888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:22.168833 16888 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:36823
I20260806 10:58:22.168881 17020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:36823 every 8 connection(s)
I20260806 10:58:22.170341 16888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:58:22.173296 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 16888
I20260806 10:58:22.173535 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:58:22.174031 31592 external_mini_cluster.cc:1428] Setting key 9ef2ed011cf4f3f7ea801dc860651f7c
Aug 06 10:58:22 dist-test-slave-49h3 krb5kdc[16637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013902, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:58:22.192677 17023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:22.173077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:59627 (local address 127.30.218.62:45655)
0806 10:58:22.173650 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:22.173657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:22.174734 (+ 1077us) server_negotiation.cc:407] Connection header received
0806 10:58:22.176757 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:22.176763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:22.176854 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:22.176951 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:22.178711 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:22.179494 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:22.181294 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:22.181819 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:22.181941 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:22.185792 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:22.185798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:22.185801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:22.185840 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:22.188848 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:22.189882 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:22.189887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:22.189890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:22.190044 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:22.190536 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:22.190540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:22.190542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:22.190848 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:22.191023 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:22.191955 (+ 932us) server_negotiation.cc:299] Negotiation successful
0806 10:58:22.192247 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":270,"threads_started":1}
I20260806 10:58:22.193840 17024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:22.173702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:45655 (local address 127.30.218.2:59627)
0806 10:58:22.174532 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:22.174582 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:22.176452 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:22.177233 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:22.177243 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:22.177883 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:58:22.178515 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:22.178536 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:22.179669 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:22.179673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:22.181128 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:22.181144 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:22.182078 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:22.182086 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:22.182338 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:22.183034 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:58:22.183065 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:58:22.185497 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:58:22.189092 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:58:22.189101 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:58:22.189119 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:58:22.189726 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:58:22.190175 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:58:22.190179 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:58:22.190181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:58:22.190406 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:58:22.190992 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:58:22.191001 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:58:22.191429 (+ 428us) client_negotiation.cc:769] Sending connection context
0806 10:58:22.191809 (+ 380us) client_negotiation.cc:240] Negotiation successful
0806 10:58:22.192438 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":342,"threads_started":1}
I20260806 10:58:22.195525 17021 heartbeater.cc:347] Connected to a master server at 127.30.218.62:45655
I20260806 10:58:22.195921 17021 heartbeater.cc:464] Registering TS with master...
I20260806 10:58:22.196897 17021 heartbeater.cc:511] Master 127.30.218.62:45655 requested a full tablet report, sending...
I20260806 10:58:22.198812 16685 ts_manager.cc:194] Registered new tserver with Master: 4cd782cd07cd4b7193abaf9a8208e214 (127.30.218.2:36823)
I20260806 10:58:22.199694 16685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.2@KRBTEST.COM'} at 127.30.218.2:59627
WARNING: no policy specified for kudu/127.30.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
I20260806 10:58:22.244438 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:45655
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:58:22.597724 17027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:22.598045 17027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:22.598142 17027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:22.605872 17027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:22.606058 17027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:22.606258 17027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:58:22.618443 17027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:45655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:22.619880 17027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:22.621176 17027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:22.635541 17033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:22.635576 17034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:22.635604 17036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:22.642388 17027 server_base.cc:1034] running on GCE node
I20260806 10:58:22.643182 16881 heartbeater.cc:503] Master 127.30.218.62:45655 was elected leader, sending a full tablet report...
I20260806 10:58:22.643270 17027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:22.644459 17027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:22.645690 17027 hybrid_clock.cc:648] HybridClock initialized: now 1786013902645643 us; error 42 us; skew 500 ppm
I20260806 10:58:22.646044 17027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:58:22 dist-test-slave-49h3 krb5kdc[16637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013902, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:58:22.649382 17027 init.cc:377] Logged in from keytab as kudu/127.30.218.3@KRBTEST.COM (short username kudu)
I20260806 10:58:22.650985 17027 webserver.cc:460] Webserver started at http://127.30.218.3:46001/ using document root <none> and password file <none>
I20260806 10:58:22.651815 17027 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:22.651924 17027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:22.652223 17027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:22.655432 17027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "e1b654539b4045faa8869bb32b7c03f0"
format_stamp: "Formatted at 2026-08-06 10:58:22 on dist-test-slave-49h3"
server_key: "25a6a148163dc897fb3be0ea81064010"
server_key_iv: "d4ffeb0e8de41282d05ac487520fe37f"
server_key_version: "encryptionkey@0"
I20260806 10:58:22.656200 17027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "e1b654539b4045faa8869bb32b7c03f0"
format_stamp: "Formatted at 2026-08-06 10:58:22 on dist-test-slave-49h3"
server_key: "25a6a148163dc897fb3be0ea81064010"
server_key_iv: "d4ffeb0e8de41282d05ac487520fe37f"
server_key_version: "encryptionkey@0"
I20260806 10:58:22.662006 17027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260806 10:58:22.665907 17043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:22.667414 17027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:58:22.667622 17027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "e1b654539b4045faa8869bb32b7c03f0"
format_stamp: "Formatted at 2026-08-06 10:58:22 on dist-test-slave-49h3"
server_key: "25a6a148163dc897fb3be0ea81064010"
server_key_iv: "d4ffeb0e8de41282d05ac487520fe37f"
server_key_version: "encryptionkey@0"
I20260806 10:58:22.667840 17027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:22.687685 17027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:22.692296 17027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:22.692625 17027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:22.693799 17027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:58:22.695340 17027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:58:22.695423 17027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:22.695567 17027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:58:22.695619 17027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:22.727378 17027 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:33201
I20260806 10:58:22.727420 17159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:33201 every 8 connection(s)
I20260806 10:58:22.728744 17027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:58:22.738232 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 17027
I20260806 10:58:22.738446 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:58:22.739017 31592 external_mini_cluster.cc:1428] Setting key 0f8c8b623c17e2bdd111cac0ab2c6a3a
Aug 06 10:58:22 dist-test-slave-49h3 krb5kdc[16637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013902, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:58:22.750376 17162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:22.731360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:58563 (local address 127.30.218.62:45655)
0806 10:58:22.731847 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:22.731852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:22.732766 (+ 914us) server_negotiation.cc:407] Connection header received
0806 10:58:22.734675 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:22.734681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:22.734764 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:22.734865 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:22.736423 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:22.737241 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:22.739469 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:22.740020 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:22.740154 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:22.744207 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:22.744212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:22.744216 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:22.744254 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:22.746997 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:22.747911 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:22.747915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:22.747918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:22.748054 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:22.748594 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:22.748598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:22.748600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:22.748903 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:22.749026 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:22.749865 (+ 839us) server_negotiation.cc:299] Negotiation successful
0806 10:58:22.750075 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":241,"threads_started":1}
I20260806 10:58:22.751395 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:22.731897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:45655 (local address 127.30.218.3:58563)
0806 10:58:22.732601 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:22.732638 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:22.734386 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:22.735128 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:22.735137 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:22.735661 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:58:22.736236 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:22.736254 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:22.737403 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:22.737408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:22.739293 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:22.739311 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:22.740273 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:22.740281 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:22.740606 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:22.741421 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:58:22.741481 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:58:22.743971 (+ 2490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:58:22.747192 (+ 3221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:58:22.747201 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:58:22.747222 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:58:22.747759 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:58:22.748209 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:58:22.748213 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:58:22.748215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:58:22.748452 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:58:22.749002 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:58:22.749008 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:58:22.749381 (+ 373us) client_negotiation.cc:769] Sending connection context
0806 10:58:22.749725 (+ 344us) client_negotiation.cc:240] Negotiation successful
0806 10:58:22.750202 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":267,"threads_started":1}
I20260806 10:58:22.752933 17160 heartbeater.cc:347] Connected to a master server at 127.30.218.62:45655
I20260806 10:58:22.753254 17160 heartbeater.cc:464] Registering TS with master...
I20260806 10:58:22.754104 17160 heartbeater.cc:511] Master 127.30.218.62:45655 requested a full tablet report, sending...
I20260806 10:58:22.755623 16685 ts_manager.cc:194] Registered new tserver with Master: e1b654539b4045faa8869bb32b7c03f0 (127.30.218.3:33201)
I20260806 10:58:22.756436 16685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.3@KRBTEST.COM'} at 127.30.218.3:58563
I20260806 10:58:22.767606 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:58:22.785429 17162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:22.772708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41420 (local address 127.30.218.62:45655)
0806 10:58:22.772925 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:22.772933 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:22.773313 (+ 380us) server_negotiation.cc:407] Connection header received
0806 10:58:22.773485 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:22.773489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:22.773563 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:22.773656 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:22.775028 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:22.775773 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:22.777120 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:22.777568 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:22.777675 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:22.780177 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:22.780190 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:22.780193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:22.780237 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:22.782924 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:22.783690 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:22.783693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:22.783695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:22.783816 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:22.784114 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:22.784117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:22.784119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:22.784457 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:22.784663 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:22.784944 (+ 281us) server_negotiation.cc:299] Negotiation successful
0806 10:58:22.785133 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 10:58:22.789804 16685 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:58:22.797389 16685 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:58:22.834200 17178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:22.810639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:36823 (local address 127.0.0.1:58262)
0806 10:58:22.823400 (+ 12761us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:22.823428 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:22.823628 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:22.825432 (+ 1804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:22.825441 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:22.825499 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:22.825970 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:22.825986 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:22.828196 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:22.828202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:22.831017 (+ 2815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:22.831035 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:22.833112 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:22.833119 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:22.833342 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:22.833408 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 10:58:22.833575 (+ 167us) client_negotiation.cc:240] Negotiation successful
0806 10:58:22.833789 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12580,"thread_start_us":15508,"threads_started":1}
I20260806 10:58:22.834228 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:22.810644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58262 (local address 127.30.218.2:36823)
0806 10:58:22.824582 (+ 13938us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:22.824593 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:22.824632 (+ 39us) server_negotiation.cc:407] Connection header received
0806 10:58:22.824767 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:22.824777 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:22.825145 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:22.825408 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:22.826431 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:22.828039 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:22.831297 (+ 3258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:22.832865 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:22.833019 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:22.833246 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:22.833687 (+ 441us) server_negotiation.cc:299] Negotiation successful
0806 10:58:22.833876 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13719,"thread_start_us":15724,"threads_started":1}
I20260806 10:58:22.835031 17174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:22.810127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:33201 (local address 127.0.0.1:52492)
0806 10:58:22.812020 (+ 1893us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:22.812075 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:22.812287 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:22.827848 (+ 15561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:22.827854 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:22.827883 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:22.828368 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:22.828382 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:22.830428 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:22.830434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:22.832319 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:22.832333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:22.834220 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:22.834226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:22.834403 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:22.834438 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 10:58:22.834522 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 10:58:22.834647 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1589,"mutex_wait_us":74,"thread_start_us":1397,"threads_started":1}
I20260806 10:58:22.835848 17175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:22.810464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52492 (local address 127.30.218.3:33201)
0806 10:58:22.827126 (+ 16662us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:22.827136 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:22.827171 (+ 35us) server_negotiation.cc:407] Connection header received
0806 10:58:22.827272 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:22.827281 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:22.827599 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:22.827826 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:22.828600 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:22.830258 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:22.832494 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:22.833727 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:22.833847 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:22.834072 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:22.835340 (+ 1268us) server_negotiation.cc:299] Negotiation successful
0806 10:58:22.835514 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16460,"thread_start_us":16295,"threads_started":1}
I20260806 10:58:22.836206 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:22.813648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34008 (local address 127.30.218.1:38647)
0806 10:58:22.827568 (+ 13920us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:22.827574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:22.827599 (+ 25us) server_negotiation.cc:407] Connection header received
0806 10:58:22.827681 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:22.827686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:22.827906 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:22.828115 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:22.828924 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:22.830683 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:22.833277 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:22.834672 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:22.834808 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:22.835031 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:22.835538 (+ 507us) server_negotiation.cc:299] Negotiation successful
0806 10:58:22.835717 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13749,"thread_start_us":13560,"threads_started":1}
I20260806 10:58:22.836715 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:22.812954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:38647 (local address 127.0.0.1:34008)
0806 10:58:22.826174 (+ 13220us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:22.826194 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:22.826309 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:22.828162 (+ 1853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:22.828167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:22.828200 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:22.828710 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:22.828726 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:22.830877 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:22.830883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:22.833053 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:22.833068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:22.835196 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:22.835202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:22.835365 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:22.835388 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 10:58:22.835469 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 10:58:22.835602 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13050,"mutex_wait_us":10,"thread_start_us":10062,"threads_started":1}
I20260806 10:58:22.838716 16956 tablet_service.cc:1467] Processing CreateTablet for tablet dc81e1e3dbbb45cb867db3f03f257cd8 (DEFAULT_TABLE table=test-table [id=f7a90797d0484152baf2c6d8ae4399fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:58:22.839197 17094 tablet_service.cc:1467] Processing CreateTablet for tablet dc81e1e3dbbb45cb867db3f03f257cd8 (DEFAULT_TABLE table=test-table [id=f7a90797d0484152baf2c6d8ae4399fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:58:22.840193 16956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc81e1e3dbbb45cb867db3f03f257cd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:22.840449 17094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc81e1e3dbbb45cb867db3f03f257cd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:22.843001 16813 tablet_service.cc:1467] Processing CreateTablet for tablet dc81e1e3dbbb45cb867db3f03f257cd8 (DEFAULT_TABLE table=test-table [id=f7a90797d0484152baf2c6d8ae4399fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:58:22.844559 16813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc81e1e3dbbb45cb867db3f03f257cd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:22.852221 17180 tablet_bootstrap.cc:492] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214: Bootstrap starting.
I20260806 10:58:22.855470 17181 tablet_bootstrap.cc:492] T dc81e1e3dbbb45cb867db3f03f257cd8 P e1b654539b4045faa8869bb32b7c03f0: Bootstrap starting.
I20260806 10:58:22.857230 17180 tablet_bootstrap.cc:654] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:22.858688 17180 log.cc:824] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:22.859261 17182 tablet_bootstrap.cc:492] T dc81e1e3dbbb45cb867db3f03f257cd8 P c23a95014736428f8329c83ebdec5ceb: Bootstrap starting.
I20260806 10:58:22.859395 17181 tablet_bootstrap.cc:654] T dc81e1e3dbbb45cb867db3f03f257cd8 P e1b654539b4045faa8869bb32b7c03f0: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:22.861433 17181 log.cc:824] T dc81e1e3dbbb45cb867db3f03f257cd8 P e1b654539b4045faa8869bb32b7c03f0: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:22.862649 17182 tablet_bootstrap.cc:654] T dc81e1e3dbbb45cb867db3f03f257cd8 P c23a95014736428f8329c83ebdec5ceb: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:22.864017 17182 log.cc:824] T dc81e1e3dbbb45cb867db3f03f257cd8 P c23a95014736428f8329c83ebdec5ceb: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:22.865114 17181 tablet_bootstrap.cc:492] T dc81e1e3dbbb45cb867db3f03f257cd8 P e1b654539b4045faa8869bb32b7c03f0: No bootstrap required, opened a new log
I20260806 10:58:22.865159 17180 tablet_bootstrap.cc:492] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214: No bootstrap required, opened a new log
I20260806 10:58:22.865556 17180 ts_tablet_manager.cc:1397] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260806 10:58:22.865572 17181 ts_tablet_manager.cc:1397] T dc81e1e3dbbb45cb867db3f03f257cd8 P e1b654539b4045faa8869bb32b7c03f0: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260806 10:58:22.868096 17182 tablet_bootstrap.cc:492] T dc81e1e3dbbb45cb867db3f03f257cd8 P c23a95014736428f8329c83ebdec5ceb: No bootstrap required, opened a new log
I20260806 10:58:22.868433 17182 ts_tablet_manager.cc:1397] T dc81e1e3dbbb45cb867db3f03f257cd8 P c23a95014736428f8329c83ebdec5ceb: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260806 10:58:22.873790 17181 raft_consensus.cc:348] T dc81e1e3dbbb45cb867db3f03f257cd8 P e1b654539b4045faa8869bb32b7c03f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1b654539b4045faa8869bb32b7c03f0" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 33201 } } peers { permanent_uuid: "4cd782cd07cd4b7193abaf9a8208e214" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 36823 } } peers { permanent_uuid: "c23a95014736428f8329c83ebdec5ceb" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38647 } }
I20260806 10:58:22.874231 17181 raft_consensus.cc:374] T dc81e1e3dbbb45cb867db3f03f257cd8 P e1b654539b4045faa8869bb32b7c03f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:22.874349 17181 raft_consensus.cc:729] T dc81e1e3dbbb45cb867db3f03f257cd8 P e1b654539b4045faa8869bb32b7c03f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1b654539b4045faa8869bb32b7c03f0, State: Initialized, Role: FOLLOWER
I20260806 10:58:22.875018 17181 consensus_queue.cc:260] T dc81e1e3dbbb45cb867db3f03f257cd8 P e1b654539b4045faa8869bb32b7c03f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1b654539b4045faa8869bb32b7c03f0" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 33201 } } peers { permanent_uuid: "4cd782cd07cd4b7193abaf9a8208e214" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 36823 } } peers { permanent_uuid: "c23a95014736428f8329c83ebdec5ceb" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38647 } }
I20260806 10:58:22.876212 17160 heartbeater.cc:503] Master 127.30.218.62:45655 was elected leader, sending a full tablet report...
I20260806 10:58:22.876365 17180 raft_consensus.cc:348] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1b654539b4045faa8869bb32b7c03f0" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 33201 } } peers { permanent_uuid: "4cd782cd07cd4b7193abaf9a8208e214" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 36823 } } peers { permanent_uuid: "c23a95014736428f8329c83ebdec5ceb" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38647 } }
I20260806 10:58:22.876837 17180 raft_consensus.cc:374] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:22.876928 17180 raft_consensus.cc:729] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd782cd07cd4b7193abaf9a8208e214, State: Initialized, Role: FOLLOWER
I20260806 10:58:22.876950 17181 ts_tablet_manager.cc:1428] T dc81e1e3dbbb45cb867db3f03f257cd8 P e1b654539b4045faa8869bb32b7c03f0: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260806 10:58:22.877674 17180 consensus_queue.cc:260] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1b654539b4045faa8869bb32b7c03f0" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 33201 } } peers { permanent_uuid: "4cd782cd07cd4b7193abaf9a8208e214" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 36823 } } peers { permanent_uuid: "c23a95014736428f8329c83ebdec5ceb" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38647 } }
I20260806 10:58:22.879675 17182 raft_consensus.cc:348] T dc81e1e3dbbb45cb867db3f03f257cd8 P c23a95014736428f8329c83ebdec5ceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1b654539b4045faa8869bb32b7c03f0" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 33201 } } peers { permanent_uuid: "4cd782cd07cd4b7193abaf9a8208e214" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 36823 } } peers { permanent_uuid: "c23a95014736428f8329c83ebdec5ceb" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38647 } }
I20260806 10:58:22.880280 17182 raft_consensus.cc:374] T dc81e1e3dbbb45cb867db3f03f257cd8 P c23a95014736428f8329c83ebdec5ceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:22.880458 17182 raft_consensus.cc:729] T dc81e1e3dbbb45cb867db3f03f257cd8 P c23a95014736428f8329c83ebdec5ceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c23a95014736428f8329c83ebdec5ceb, State: Initialized, Role: FOLLOWER
I20260806 10:58:22.881198 17182 consensus_queue.cc:260] T dc81e1e3dbbb45cb867db3f03f257cd8 P c23a95014736428f8329c83ebdec5ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1b654539b4045faa8869bb32b7c03f0" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 33201 } } peers { permanent_uuid: "4cd782cd07cd4b7193abaf9a8208e214" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 36823 } } peers { permanent_uuid: "c23a95014736428f8329c83ebdec5ceb" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38647 } }
I20260806 10:58:22.881892 17021 heartbeater.cc:503] Master 127.30.218.62:45655 was elected leader, sending a full tablet report...
I20260806 10:58:22.881964 17180 ts_tablet_manager.cc:1428] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214: Time spent starting tablet: real 0.016s user 0.010s sys 0.006s
I20260806 10:58:22.887319 17182 ts_tablet_manager.cc:1428] T dc81e1e3dbbb45cb867db3f03f257cd8 P c23a95014736428f8329c83ebdec5ceb: Time spent starting tablet: real 0.019s user 0.012s sys 0.009s
W20260806 10:58:22.922487 17022 tablet.cc:2369] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:58:22.980404 17161 tablet.cc:2369] T dc81e1e3dbbb45cb867db3f03f257cd8 P e1b654539b4045faa8869bb32b7c03f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:58:23.057430 17187 raft_consensus.cc:482] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:58:23.057783 17187 raft_consensus.cc:504] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1b654539b4045faa8869bb32b7c03f0" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 33201 } } peers { permanent_uuid: "4cd782cd07cd4b7193abaf9a8208e214" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 36823 } } peers { permanent_uuid: "c23a95014736428f8329c83ebdec5ceb" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38647 } }
I20260806 10:58:23.059589 17187 leader_election.cc:288] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1b654539b4045faa8869bb32b7c03f0 (127.30.218.3:33201), c23a95014736428f8329c83ebdec5ceb (127.30.218.1:38647)
I20260806 10:58:23.066252 17175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:23.060004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:46543 (local address 127.30.218.3:33201)
0806 10:58:23.060199 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:23.060203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:23.060811 (+ 608us) server_negotiation.cc:407] Connection header received
0806 10:58:23.061012 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:23.061016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:23.061081 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:23.061164 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:23.061873 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:23.062647 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:23.064459 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:23.065264 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:23.065428 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:23.065547 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:23.065834 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 10:58:23.065973 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 10:58:23.066329 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:23.059999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:33201 (local address 127.30.218.2:46543)
0806 10:58:23.060668 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:23.060686 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:23.060870 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:23.061199 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:23.061204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:23.061232 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:23.061704 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:23.061724 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:23.062787 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:23.062791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:23.064287 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:23.064301 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:23.065520 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:23.065527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:23.065670 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:23.065695 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 10:58:23.065789 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 10:58:23.065935 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":349,"threads_started":1}
I20260806 10:58:23.066982 17190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:23.060856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:38647 (local address 127.30.218.2:49745)
0806 10:58:23.061743 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:23.061759 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:23.061887 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:23.062221 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:23.062226 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:23.062252 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:23.062550 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:23.062559 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:23.063585 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:23.063589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:23.064970 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:23.064984 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:23.066264 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:23.066270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:23.066473 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:23.066508 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 10:58:23.066599 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 10:58:23.066732 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":512,"threads_started":1}
I20260806 10:58:23.067018 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:23.061107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:49745 (local address 127.30.218.1:38647)
0806 10:58:23.061512 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:23.061519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:23.061888 (+ 369us) server_negotiation.cc:407] Connection header received
0806 10:58:23.062032 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:23.062036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:23.062100 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:23.062179 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:23.062716 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:23.063437 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:23.065148 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:23.066037 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:23.066198 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:23.066272 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:23.066643 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 10:58:23.066788 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207}
I20260806 10:58:23.067546 17114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc81e1e3dbbb45cb867db3f03f257cd8" candidate_uuid: "4cd782cd07cd4b7193abaf9a8208e214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1b654539b4045faa8869bb32b7c03f0" is_pre_election: true
I20260806 10:58:23.068185 17114 raft_consensus.cc:2454] T dc81e1e3dbbb45cb867db3f03f257cd8 P e1b654539b4045faa8869bb32b7c03f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd782cd07cd4b7193abaf9a8208e214 in term 0.
I20260806 10:58:23.068120 16834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc81e1e3dbbb45cb867db3f03f257cd8" candidate_uuid: "4cd782cd07cd4b7193abaf9a8208e214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c23a95014736428f8329c83ebdec5ceb" is_pre_election: true
I20260806 10:58:23.068750 16834 raft_consensus.cc:2454] T dc81e1e3dbbb45cb867db3f03f257cd8 P c23a95014736428f8329c83ebdec5ceb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd782cd07cd4b7193abaf9a8208e214 in term 0.
I20260806 10:58:23.069057 16907 leader_election.cc:302] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd782cd07cd4b7193abaf9a8208e214, e1b654539b4045faa8869bb32b7c03f0; no voters:
I20260806 10:58:23.069551 17187 raft_consensus.cc:2781] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:58:23.069739 17187 raft_consensus.cc:482] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:58:23.069885 17187 raft_consensus.cc:3037] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:23.072464 17187 raft_consensus.cc:504] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1b654539b4045faa8869bb32b7c03f0" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 33201 } } peers { permanent_uuid: "4cd782cd07cd4b7193abaf9a8208e214" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 36823 } } peers { permanent_uuid: "c23a95014736428f8329c83ebdec5ceb" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38647 } }
I20260806 10:58:23.073155 17187 leader_election.cc:288] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214 [CANDIDATE]: Term 1 election: Requested vote from peers e1b654539b4045faa8869bb32b7c03f0 (127.30.218.3:33201), c23a95014736428f8329c83ebdec5ceb (127.30.218.1:38647)
I20260806 10:58:23.073772 17114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc81e1e3dbbb45cb867db3f03f257cd8" candidate_uuid: "4cd782cd07cd4b7193abaf9a8208e214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1b654539b4045faa8869bb32b7c03f0"
I20260806 10:58:23.073848 16834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc81e1e3dbbb45cb867db3f03f257cd8" candidate_uuid: "4cd782cd07cd4b7193abaf9a8208e214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c23a95014736428f8329c83ebdec5ceb"
I20260806 10:58:23.073994 17114 raft_consensus.cc:3037] T dc81e1e3dbbb45cb867db3f03f257cd8 P e1b654539b4045faa8869bb32b7c03f0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:23.074062 16834 raft_consensus.cc:3037] T dc81e1e3dbbb45cb867db3f03f257cd8 P c23a95014736428f8329c83ebdec5ceb [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:23.076323 17114 raft_consensus.cc:2454] T dc81e1e3dbbb45cb867db3f03f257cd8 P e1b654539b4045faa8869bb32b7c03f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd782cd07cd4b7193abaf9a8208e214 in term 1.
I20260806 10:58:23.076344 16834 raft_consensus.cc:2454] T dc81e1e3dbbb45cb867db3f03f257cd8 P c23a95014736428f8329c83ebdec5ceb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd782cd07cd4b7193abaf9a8208e214 in term 1.
I20260806 10:58:23.076808 16907 leader_election.cc:302] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd782cd07cd4b7193abaf9a8208e214, e1b654539b4045faa8869bb32b7c03f0; no voters:
I20260806 10:58:23.077116 17187 raft_consensus.cc:2781] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:58:23.077642 17187 raft_consensus.cc:686] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd782cd07cd4b7193abaf9a8208e214, State: Running, Role: LEADER
I20260806 10:58:23.078428 17187 consensus_queue.cc:237] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1b654539b4045faa8869bb32b7c03f0" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 33201 } } peers { permanent_uuid: "4cd782cd07cd4b7193abaf9a8208e214" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 36823 } } peers { permanent_uuid: "c23a95014736428f8329c83ebdec5ceb" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38647 } }
I20260806 10:58:23.083606 16684 catalog_manager.cc:5495] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd782cd07cd4b7193abaf9a8208e214 (127.30.218.2). New cstate: current_term: 1 leader_uuid: "4cd782cd07cd4b7193abaf9a8208e214" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1b654539b4045faa8869bb32b7c03f0" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 33201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cd782cd07cd4b7193abaf9a8208e214" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 36823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c23a95014736428f8329c83ebdec5ceb" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38647 } health_report { overall_health: UNKNOWN } } }
W20260806 10:58:23.110594 16882 tablet.cc:2369] T dc81e1e3dbbb45cb867db3f03f257cd8 P c23a95014736428f8329c83ebdec5ceb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:58:23.123278 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:23.118111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58276 (local address 127.30.218.2:36823)
0806 10:58:23.118289 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:23.118293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:23.118312 (+ 19us) server_negotiation.cc:407] Connection header received
0806 10:58:23.118464 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:23.118468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:23.118552 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:23.118692 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:58:23.119006 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:23.119758 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:23.121141 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:23.121618 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:23.121733 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:23.122170 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:58:23.122734 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:58:23.122856 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:23.122966 (+ 110us) server_negotiation.cc:299] Negotiation successful
0806 10:58:23.123046 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 10:58:23.132620 16834 raft_consensus.cc:1261] T dc81e1e3dbbb45cb867db3f03f257cd8 P c23a95014736428f8329c83ebdec5ceb [term 1 FOLLOWER]: Refusing update from remote peer 4cd782cd07cd4b7193abaf9a8208e214: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:58:23.132614 17114 raft_consensus.cc:1261] T dc81e1e3dbbb45cb867db3f03f257cd8 P e1b654539b4045faa8869bb32b7c03f0 [term 1 FOLLOWER]: Refusing update from remote peer 4cd782cd07cd4b7193abaf9a8208e214: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:58:23.133859 17191 consensus_queue.cc:1035] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1b654539b4045faa8869bb32b7c03f0" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 33201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:58:23.134250 17187 consensus_queue.cc:1035] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c23a95014736428f8329c83ebdec5ceb" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:58:23.170076 16685 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:58:23.170447 16685 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:58:23.174619 16685 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f2335aa7d6240e092d45672e3cd1bd0: Sending DeleteTablet for 3 replicas of tablet dc81e1e3dbbb45cb867db3f03f257cd8
I20260806 10:58:23.175906 17094 tablet_service.cc:1514] Processing DeleteTablet for tablet dc81e1e3dbbb45cb867db3f03f257cd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:58:23 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:52492
I20260806 10:58:23.175881 16956 tablet_service.cc:1514] Processing DeleteTablet for tablet dc81e1e3dbbb45cb867db3f03f257cd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:58:23 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:58262
I20260806 10:58:23.176759 17204 tablet_replica.cc:330] T dc81e1e3dbbb45cb867db3f03f257cd8 P e1b654539b4045faa8869bb32b7c03f0: stopping tablet replica
I20260806 10:58:23.177156 16813 tablet_service.cc:1514] Processing DeleteTablet for tablet dc81e1e3dbbb45cb867db3f03f257cd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:58:23 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:34008
I20260806 10:58:23.177307 17204 raft_consensus.cc:2229] T dc81e1e3dbbb45cb867db3f03f257cd8 P e1b654539b4045faa8869bb32b7c03f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:58:23.177888 17204 raft_consensus.cc:2258] T dc81e1e3dbbb45cb867db3f03f257cd8 P e1b654539b4045faa8869bb32b7c03f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:58:23.179301 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 16748
I20260806 10:58:23.189100 17206 tablet_replica.cc:330] T dc81e1e3dbbb45cb867db3f03f257cd8 P c23a95014736428f8329c83ebdec5ceb: stopping tablet replica
I20260806 10:58:23.189531 17205 tablet_replica.cc:330] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214: stopping tablet replica
I20260806 10:58:23.191040 17205 raft_consensus.cc:2229] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:58:23.192157 17205 raft_consensus.cc:2258] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:58:23.192375 17206 raft_consensus.cc:2229] T dc81e1e3dbbb45cb867db3f03f257cd8 P c23a95014736428f8329c83ebdec5ceb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:58:23.193023 17206 raft_consensus.cc:2258] T dc81e1e3dbbb45cb867db3f03f257cd8 P c23a95014736428f8329c83ebdec5ceb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:58:23.193856 17204 ts_tablet_manager.cc:1905] T dc81e1e3dbbb45cb867db3f03f257cd8 P e1b654539b4045faa8869bb32b7c03f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:58:23.194864 17205 ts_tablet_manager.cc:1905] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:58:23.198890 17206 ts_tablet_manager.cc:1905] T dc81e1e3dbbb45cb867db3f03f257cd8 P c23a95014736428f8329c83ebdec5ceb: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 10:58:23 dist-test-slave-49h3 krb5kdc[16637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.1@KRBTEST.COM
I20260806 10:58:23.205142 17205 ts_tablet_manager.cc:1918] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:58:23.205340 17205 log.cc:1196] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/dc81e1e3dbbb45cb867db3f03f257cd8
I20260806 10:58:23.207466 17204 ts_tablet_manager.cc:1918] T dc81e1e3dbbb45cb867db3f03f257cd8 P e1b654539b4045faa8869bb32b7c03f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:58:23.207691 17204 log.cc:1196] T dc81e1e3dbbb45cb867db3f03f257cd8 P e1b654539b4045faa8869bb32b7c03f0: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/dc81e1e3dbbb45cb867db3f03f257cd8
I20260806 10:58:23.208287 17204 ts_tablet_manager.cc:1939] T dc81e1e3dbbb45cb867db3f03f257cd8 P e1b654539b4045faa8869bb32b7c03f0: Deleting consensus metadata
I20260806 10:58:23.210681 16670 catalog_manager.cc:4841] TS e1b654539b4045faa8869bb32b7c03f0 (127.30.218.3:33201): tablet dc81e1e3dbbb45cb867db3f03f257cd8 (table test-table [id=f7a90797d0484152baf2c6d8ae4399fb]) successfully deleted
I20260806 10:58:23.211745 17205 ts_tablet_manager.cc:1939] T dc81e1e3dbbb45cb867db3f03f257cd8 P 4cd782cd07cd4b7193abaf9a8208e214: Deleting consensus metadata
I20260806 10:58:23.211920 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:23.180479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34016 (local address 127.30.218.1:38647)
0806 10:58:23.180678 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:23.180684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:23.190357 (+ 9673us) server_negotiation.cc:407] Connection header received
0806 10:58:23.190550 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:23.190555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:23.190664 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:23.190768 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:23.193046 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:23.194255 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:23.196418 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:23.197139 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:23.197263 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:23.203381 (+ 6118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:23.203404 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:23.203425 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:23.203484 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:23.207643 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:23.208865 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:23.208877 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:23.208880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:23.209077 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:23.209706 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:23.209713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:23.209716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:23.210254 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:23.210589 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:23.211241 (+ 652us) server_negotiation.cc:299] Negotiation successful
0806 10:58:23.211501 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 10:58:23.213949 16876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:23.214289 16668 catalog_manager.cc:4841] TS 4cd782cd07cd4b7193abaf9a8208e214 (127.30.218.2:36823): tablet dc81e1e3dbbb45cb867db3f03f257cd8 (table test-table [id=f7a90797d0484152baf2c6d8ae4399fb]) successfully deleted
I20260806 10:58:23.307214 17206 ts_tablet_manager.cc:1918] T dc81e1e3dbbb45cb867db3f03f257cd8 P c23a95014736428f8329c83ebdec5ceb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:58:23.307507 17206 log.cc:1196] T dc81e1e3dbbb45cb867db3f03f257cd8 P c23a95014736428f8329c83ebdec5ceb: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786013589554914-31592-0/minicluster-data/ts-0/wal/wals/dc81e1e3dbbb45cb867db3f03f257cd8
I20260806 10:58:23.307996 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 16748
I20260806 10:58:23.308133 17206 ts_tablet_manager.cc:1939] T dc81e1e3dbbb45cb867db3f03f257cd8 P c23a95014736428f8329c83ebdec5ceb: Deleting consensus metadata
W20260806 10:58:23.325877 16670 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.218.1:38647 (error 108)
I20260806 10:58:23.327296 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 16888
I20260806 10:58:23.327806 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:23.327270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:38647 (local address 127.0.0.1:34024)
0806 10:58:23.327487 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:23.327633 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260806 10:58:23.328183 16670 catalog_manager.cc:4584] TS c23a95014736428f8329c83ebdec5ceb (127.30.218.1:38647): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc81e1e3dbbb45cb867db3f03f257cd8: Network error: Client connection negotiation failed: client connection to 127.30.218.1:38647: connect: Connection refused (error 111)
Aug 06 10:58:23 dist-test-slave-49h3 krb5kdc[16637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.2@KRBTEST.COM
I20260806 10:58:23.346308 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:23.328183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58282 (local address 127.30.218.2:36823)
0806 10:58:23.328381 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:23.328387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:23.328415 (+ 28us) server_negotiation.cc:407] Connection header received
0806 10:58:23.328596 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:23.328601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:23.328672 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:23.328765 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:23.330664 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:23.331566 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:23.333755 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:23.334278 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:23.334399 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:23.340003 (+ 5604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:23.340023 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:23.340048 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:23.340091 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:23.343293 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:23.344210 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:23.344217 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:23.344220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:23.344378 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:23.344761 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:23.344768 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:23.344770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:23.345104 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:23.345382 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:23.345778 (+ 396us) server_negotiation.cc:299] Negotiation successful
0806 10:58:23.345967 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 10:58:23.346974 17016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:23.381793 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:23.381275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:38647 (local address 127.0.0.1:34032)
0806 10:58:23.381518 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:23.381633 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 10:58:23.382946 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:23.382585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:38647 (local address 127.0.0.1:34034)
0806 10:58:23.382737 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:23.382812 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 10:58:23.443782 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:23.443231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:38647 (local address 127.0.0.1:34044)
0806 10:58:23.443486 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:23.443598 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260806 10:58:23.444912 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:23.444557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:38647 (local address 127.0.0.1:34050)
0806 10:58:23.444704 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:23.444784 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 10:58:23.464181 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 16888
I20260806 10:58:23.483516 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 17027
Aug 06 10:58:23 dist-test-slave-49h3 krb5kdc[16637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.3@KRBTEST.COM
I20260806 10:58:23.500442 17175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:23.484424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52502 (local address 127.30.218.3:33201)
0806 10:58:23.484601 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:23.484606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:23.484627 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:58:23.484758 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:23.484762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:23.484828 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:23.484919 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:23.486333 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:23.487069 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:23.488494 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:23.488896 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:23.488993 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:23.494156 (+ 5163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:23.494176 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:23.494193 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:23.494245 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:23.497297 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:23.498289 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:23.498298 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:23.498300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:23.498478 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:23.498897 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:23.498900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:23.498902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:23.499304 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:23.499539 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:23.499943 (+ 404us) server_negotiation.cc:299] Negotiation successful
0806 10:58:23.500152 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 10:58:23.501171 17155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:23.525748 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:23.525091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:38647 (local address 127.0.0.1:34064)
0806 10:58:23.525365 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:23.525521 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260806 10:58:23.526949 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:23.526480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:38647 (local address 127.0.0.1:34078)
0806 10:58:23.526711 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:23.526808 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260806 10:58:23.591526 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 17027
I20260806 10:58:23.607736 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 16651
I20260806 10:58:23.608871 16716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:23.723531 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 16651
2026-08-06T10:58:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4693 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T10:58:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:58:23Z Disabled control of system clock
I20260806 10:58:23.821313 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:34189
--webserver_interface=127.30.218.62
--webserver_port=0
--builtin_ntp_servers=127.30.218.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:34189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 10:58:24.178869 17217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:24.179210 17217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:24.179338 17217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:24.187168 17217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:58:24.187314 17217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:24.187361 17217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:24.187445 17217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:58:24.187500 17217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:58:24.197510 17217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:34189
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:34189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:24.198936 17217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:24.200944 17217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:24.214728 17226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:24.215194 17223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:24.216089 17224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:24.216212 17217 server_base.cc:1034] running on GCE node
I20260806 10:58:24.217469 17217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:24.219636 17217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:24.220870 17217 hybrid_clock.cc:648] HybridClock initialized: now 1786013904220817 us; error 51 us; skew 500 ppm
I20260806 10:58:24.221277 17217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:58:24.223865 17217 webserver.cc:460] Webserver started at http://127.30.218.62:37371/ using document root <none> and password file <none>
I20260806 10:58:24.224658 17217 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:24.224767 17217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:24.225112 17217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:24.228499 17217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "8e518b2f190a40a2a39787787800bccc"
format_stamp: "Formatted at 2026-08-06 10:58:24 on dist-test-slave-49h3"
server_key: "076dbca459c3f5289b4c3920181cceaf"
server_key_iv: "9bd5f73a6e2862a16f48b82d2ef308c6"
server_key_version: "encryptionkey@0"
I20260806 10:58:24.229429 17217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "8e518b2f190a40a2a39787787800bccc"
format_stamp: "Formatted at 2026-08-06 10:58:24 on dist-test-slave-49h3"
server_key: "076dbca459c3f5289b4c3920181cceaf"
server_key_iv: "9bd5f73a6e2862a16f48b82d2ef308c6"
server_key_version: "encryptionkey@0"
I20260806 10:58:24.235381 17217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 10:58:24.239378 17232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:24.240881 17217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 10:58:24.241084 17217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "8e518b2f190a40a2a39787787800bccc"
format_stamp: "Formatted at 2026-08-06 10:58:24 on dist-test-slave-49h3"
server_key: "076dbca459c3f5289b4c3920181cceaf"
server_key_iv: "9bd5f73a6e2862a16f48b82d2ef308c6"
server_key_version: "encryptionkey@0"
I20260806 10:58:24.241333 17217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:24.278507 17217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:24.279516 17217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:24.279827 17217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:24.298671 17217 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:34189
I20260806 10:58:24.298677 17286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:34189 every 8 connection(s)
I20260806 10:58:24.300248 17217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:58:24.304126 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 17217
I20260806 10:58:24.304320 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:58:24.304328 17287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:24.304797 31592 external_mini_cluster.cc:1428] Setting key 2d47968e73e9df02b166130a3236e485
I20260806 10:58:24.318378 17287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc: Bootstrap starting.
I20260806 10:58:24.320076 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:24.306830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35292 (local address 127.30.218.62:34189)
0806 10:58:24.308024 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:24.308035 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:24.308087 (+ 52us) server_negotiation.cc:407] Connection header received
0806 10:58:24.310501 (+ 2414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:24.310529 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:24.311111 (+ 582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:24.311476 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:24.312153 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:24.313441 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:24.315426 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:24.316190 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:24.316335 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:24.317682 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:24.317690 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:24.317713 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:24.318273 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:24.318356 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:24.318383 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:24.318474 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:24.318732 (+ 258us) server_negotiation.cc:299] Negotiation successful
0806 10:58:24.319082 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":439,"threads_started":1}
I20260806 10:58:24.322175 17287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:24.323614 17287 log.cc:824] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:24.327225 17287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc: No bootstrap required, opened a new log
I20260806 10:58:24.338235 17287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e518b2f190a40a2a39787787800bccc" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34189 } }
I20260806 10:58:24.338784 17287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:24.338910 17287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e518b2f190a40a2a39787787800bccc, State: Initialized, Role: FOLLOWER
I20260806 10:58:24.339655 17287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e518b2f190a40a2a39787787800bccc" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34189 } }
I20260806 10:58:24.339972 17287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:58:24.340145 17287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:58:24.340356 17287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:24.343961 17287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e518b2f190a40a2a39787787800bccc" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34189 } }
I20260806 10:58:24.344697 17287 leader_election.cc:302] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e518b2f190a40a2a39787787800bccc; no voters:
I20260806 10:58:24.345489 17287 leader_election.cc:288] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:58:24.345785 17292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:58:24.346673 17292 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc [term 1 LEADER]: Becoming Leader. State: Replica: 8e518b2f190a40a2a39787787800bccc, State: Running, Role: LEADER
I20260806 10:58:24.347342 17292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e518b2f190a40a2a39787787800bccc" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34189 } }
I20260806 10:58:24.347764 17287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:58:24.354033 17293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e518b2f190a40a2a39787787800bccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e518b2f190a40a2a39787787800bccc" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34189 } } }
I20260806 10:58:24.354458 17293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:24.354969 17294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e518b2f190a40a2a39787787800bccc. Latest consensus state: current_term: 1 leader_uuid: "8e518b2f190a40a2a39787787800bccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e518b2f190a40a2a39787787800bccc" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34189 } } }
I20260806 10:58:24.355288 17294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc [sys.catalog]: This master's current role is: LEADER
W20260806 10:58:24.358891 17306 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 10:58:24.359014 17306 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 10:58:24.365553 17307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:58:24.367871 17307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:58:24.378715 17307 catalog_manager.cc:1347] Generated new cluster ID: aac3690a2b1f49ba9e57a38a1d3b5237
I20260806 10:58:24.378850 17307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:58:24.387252 17307 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:58:24.388659 17307 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:58:24.398312 17307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc: Generated new TSK 0
I20260806 10:58:24.399289 17307 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:58:24.413738 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:34189
--builtin_ntp_servers=127.30.218.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 10:58:24.799217 17311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:24.799543 17311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:24.799662 17311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:24.807888 17311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:24.808037 17311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:24.808216 17311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:58:24.818468 17311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:34189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:24.820127 17311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:24.822147 17311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:24.836313 17316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:24.836246 17319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:24.836203 17317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:24.837278 17311 server_base.cc:1034] running on GCE node
I20260806 10:58:24.837967 17311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:24.839027 17311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:24.840276 17311 hybrid_clock.cc:648] HybridClock initialized: now 1786013904840224 us; error 56 us; skew 500 ppm
I20260806 10:58:24.840644 17311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:58:24.843202 17311 webserver.cc:460] Webserver started at http://127.30.218.1:33047/ using document root <none> and password file <none>
I20260806 10:58:24.843961 17311 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:24.844111 17311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:24.844439 17311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:24.847795 17311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "445f77820d504541b0e56297ac3a7358"
format_stamp: "Formatted at 2026-08-06 10:58:24 on dist-test-slave-49h3"
server_key: "badfe2eb72467c8b57ee37363b9b6313"
server_key_iv: "def94f3944099f0f96f9c91c08efa9c5"
server_key_version: "encryptionkey@0"
I20260806 10:58:24.848785 17311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "445f77820d504541b0e56297ac3a7358"
format_stamp: "Formatted at 2026-08-06 10:58:24 on dist-test-slave-49h3"
server_key: "badfe2eb72467c8b57ee37363b9b6313"
server_key_iv: "def94f3944099f0f96f9c91c08efa9c5"
server_key_version: "encryptionkey@0"
I20260806 10:58:24.854851 17311 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260806 10:58:24.859401 17326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:24.860909 17311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 10:58:24.861099 17311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "445f77820d504541b0e56297ac3a7358"
format_stamp: "Formatted at 2026-08-06 10:58:24 on dist-test-slave-49h3"
server_key: "badfe2eb72467c8b57ee37363b9b6313"
server_key_iv: "def94f3944099f0f96f9c91c08efa9c5"
server_key_version: "encryptionkey@0"
I20260806 10:58:24.861351 17311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:24.883247 17311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:24.884147 17311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:24.884462 17311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:24.885541 17311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:58:24.887123 17311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:58:24.887223 17311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:24.887326 17311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:58:24.887396 17311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:24.920008 17311 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:38627
I20260806 10:58:24.920077 17442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:38627 every 8 connection(s)
I20260806 10:58:24.921427 17311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:58:24.925904 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 17311
I20260806 10:58:24.926093 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:58:24.926558 31592 external_mini_cluster.cc:1428] Setting key 90f5c8c1586c56a17dc41d1c11b14939
I20260806 10:58:24.929402 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:34189
--builtin_ntp_servers=127.30.218.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 10:58:24.934707 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:24.924080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:50599 (local address 127.30.218.62:34189)
0806 10:58:24.924553 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:24.924557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:24.925725 (+ 1168us) server_negotiation.cc:407] Connection header received
0806 10:58:24.927795 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:24.927800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:24.927887 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:24.927983 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:24.929208 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:24.929990 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:24.931656 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:24.932074 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:24.932199 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:24.933679 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:24.933685 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:24.933688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:24.933710 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:24.933754 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:24.933766 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:24.933846 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:24.934083 (+ 237us) server_negotiation.cc:299] Negotiation successful
0806 10:58:24.934311 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":245,"threads_started":1}
I20260806 10:58:24.935880 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:24.924567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34189 (local address 127.30.218.1:50599)
0806 10:58:24.925553 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:24.925593 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:24.927492 (+ 1899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:24.928268 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:24.928280 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:24.928396 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:58:24.929028 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:24.929045 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:24.930151 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:24.930154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:24.931491 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:24.931505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:24.932324 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:24.932332 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:24.932578 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:24.933297 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:58:24.933299 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:58:24.933346 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:58:24.933467 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:58:24.933860 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:58:24.933873 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:58:24.933883 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 10:58:24.934056 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 10:58:24.934613 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":273,"threads_started":1}
I20260806 10:58:24.937757 17443 heartbeater.cc:347] Connected to a master server at 127.30.218.62:34189
I20260806 10:58:24.938095 17443 heartbeater.cc:464] Registering TS with master...
I20260806 10:58:24.938917 17443 heartbeater.cc:511] Master 127.30.218.62:34189 requested a full tablet report, sending...
I20260806 10:58:24.941056 17250 ts_manager.cc:194] Registered new tserver with Master: 445f77820d504541b0e56297ac3a7358 (127.30.218.1:38627)
I20260806 10:58:24.942619 17250 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.218.1:50599
W20260806 10:58:25.279147 17447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:25.279479 17447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:25.279593 17447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:25.287350 17447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:25.287482 17447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:25.287631 17447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:58:25.297644 17447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:34189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:25.299391 17447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:25.301924 17447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:25.317685 17453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:25.317797 17452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:25.318691 17455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:25.318974 17447 server_base.cc:1034] running on GCE node
I20260806 10:58:25.319648 17447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:25.320693 17447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:25.321923 17447 hybrid_clock.cc:648] HybridClock initialized: now 1786013905321852 us; error 60 us; skew 500 ppm
I20260806 10:58:25.322276 17447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:58:25.324757 17447 webserver.cc:460] Webserver started at http://127.30.218.2:37471/ using document root <none> and password file <none>
I20260806 10:58:25.325534 17447 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:25.325662 17447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:25.325997 17447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:25.329147 17447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "979ad97f5aab48bf8d83c7713711e720"
format_stamp: "Formatted at 2026-08-06 10:58:25 on dist-test-slave-49h3"
server_key: "5bd35a1f6f841dc85dc3ac46f3b0e314"
server_key_iv: "aac6c6bff2af3783b30cd4efe79f774c"
server_key_version: "encryptionkey@0"
I20260806 10:58:25.330045 17447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "979ad97f5aab48bf8d83c7713711e720"
format_stamp: "Formatted at 2026-08-06 10:58:25 on dist-test-slave-49h3"
server_key: "5bd35a1f6f841dc85dc3ac46f3b0e314"
server_key_iv: "aac6c6bff2af3783b30cd4efe79f774c"
server_key_version: "encryptionkey@0"
I20260806 10:58:25.335569 17447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:58:25.339568 17461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:25.341079 17447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:58:25.341269 17447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "979ad97f5aab48bf8d83c7713711e720"
format_stamp: "Formatted at 2026-08-06 10:58:25 on dist-test-slave-49h3"
server_key: "5bd35a1f6f841dc85dc3ac46f3b0e314"
server_key_iv: "aac6c6bff2af3783b30cd4efe79f774c"
server_key_version: "encryptionkey@0"
I20260806 10:58:25.341554 17447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:25.370093 17447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:25.371083 17447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:25.371397 17447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:25.372489 17447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:58:25.374137 17447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:58:25.374244 17447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:25.374362 17447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:58:25.374433 17447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:25.406883 17447 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:44033
I20260806 10:58:25.406941 17581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:44033 every 8 connection(s)
I20260806 10:58:25.408265 17447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:58:25.414641 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 17447
I20260806 10:58:25.414834 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:58:25.415421 31592 external_mini_cluster.cc:1428] Setting key 71f9703545ae37e277e9866cd99ac93e
I20260806 10:58:25.418737 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:34189
--builtin_ntp_servers=127.30.218.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 10:58:25.422190 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:25.411031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:55751 (local address 127.30.218.62:34189)
0806 10:58:25.411257 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:25.411264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:25.412746 (+ 1482us) server_negotiation.cc:407] Connection header received
0806 10:58:25.414754 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:25.414760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:25.414858 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:25.414959 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:25.416482 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:25.417352 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:25.419167 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:25.419724 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:25.419864 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:25.421313 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:25.421320 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:25.421324 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:25.421349 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:25.421393 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:25.421408 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:25.421531 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:25.421740 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 10:58:25.421865 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":69}
I20260806 10:58:25.423153 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:25.411601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34189 (local address 127.30.218.2:55751)
0806 10:58:25.412566 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:25.412611 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:25.414420 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:25.415274 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:25.415288 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:25.415425 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:58:25.416286 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:25.416316 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:25.417550 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:25.417557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:25.418999 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:25.419014 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:25.419957 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:25.419962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:25.420181 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:25.420922 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:58:25.420926 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:58:25.420980 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:58:25.421076 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:58:25.421527 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:58:25.421535 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:58:25.421542 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 10:58:25.421713 (+ 171us) client_negotiation.cc:240] Negotiation successful
0806 10:58:25.422067 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":289,"threads_started":1}
I20260806 10:58:25.424738 17582 heartbeater.cc:347] Connected to a master server at 127.30.218.62:34189
I20260806 10:58:25.425117 17582 heartbeater.cc:464] Registering TS with master...
I20260806 10:58:25.426210 17582 heartbeater.cc:511] Master 127.30.218.62:34189 requested a full tablet report, sending...
I20260806 10:58:25.428104 17250 ts_manager.cc:194] Registered new tserver with Master: 979ad97f5aab48bf8d83c7713711e720 (127.30.218.2:44033)
I20260806 10:58:25.428977 17250 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.218.2:55751
W20260806 10:58:25.773972 17585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:25.774288 17585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:25.774410 17585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:25.781965 17585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:25.782096 17585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:25.782238 17585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:58:25.792047 17585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:34189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:25.793561 17585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:25.795612 17585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:25.809731 17590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:25.809818 17593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:25.809779 17591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:25.810726 17585 server_base.cc:1034] running on GCE node
I20260806 10:58:25.811386 17585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:25.812463 17585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:25.813674 17585 hybrid_clock.cc:648] HybridClock initialized: now 1786013905813621 us; error 47 us; skew 500 ppm
I20260806 10:58:25.814054 17585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:58:25.816506 17585 webserver.cc:460] Webserver started at http://127.30.218.3:40849/ using document root <none> and password file <none>
I20260806 10:58:25.817281 17585 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:25.817396 17585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:25.817778 17585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:25.821045 17585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "6613c7d07fd041c9bb907353978e0592"
format_stamp: "Formatted at 2026-08-06 10:58:25 on dist-test-slave-49h3"
server_key: "f7208f6506c04a12c57ec52e43065f67"
server_key_iv: "ad21798e1b9493de9abdc941b5ab7704"
server_key_version: "encryptionkey@0"
I20260806 10:58:25.821962 17585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "6613c7d07fd041c9bb907353978e0592"
format_stamp: "Formatted at 2026-08-06 10:58:25 on dist-test-slave-49h3"
server_key: "f7208f6506c04a12c57ec52e43065f67"
server_key_iv: "ad21798e1b9493de9abdc941b5ab7704"
server_key_version: "encryptionkey@0"
I20260806 10:58:25.828078 17585 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260806 10:58:25.832211 17599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:25.833751 17585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 10:58:25.833945 17585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "6613c7d07fd041c9bb907353978e0592"
format_stamp: "Formatted at 2026-08-06 10:58:25 on dist-test-slave-49h3"
server_key: "f7208f6506c04a12c57ec52e43065f67"
server_key_iv: "ad21798e1b9493de9abdc941b5ab7704"
server_key_version: "encryptionkey@0"
I20260806 10:58:25.834179 17585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:25.859172 17585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:25.860157 17585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:25.860482 17585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:25.861577 17585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:58:25.863117 17585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:58:25.863209 17585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:25.863317 17585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:58:25.863387 17585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:25.894603 17585 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:42247
I20260806 10:58:25.894666 17717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:42247 every 8 connection(s)
I20260806 10:58:25.896035 17585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:58:25.905937 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 17585
I20260806 10:58:25.906138 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:58:25.906582 31592 external_mini_cluster.cc:1428] Setting key dd0aa54f2cea6038ef54ef04692c754d
I20260806 10:58:25.913664 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:25.898797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:41039 (local address 127.30.218.62:34189)
0806 10:58:25.898991 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:25.898995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:25.900489 (+ 1494us) server_negotiation.cc:407] Connection header received
0806 10:58:25.903128 (+ 2639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:25.903136 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:25.903239 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:25.903330 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:25.904657 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:25.905437 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:25.907357 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:25.907890 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:25.908017 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:25.912821 (+ 4804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:25.912827 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:25.912831 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:25.912855 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:25.912898 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:25.912912 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:25.912992 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:25.913232 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 10:58:25.913337 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 10:58:25.914863 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:25.899325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:34189 (local address 127.30.218.3:41039)
0806 10:58:25.900290 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:25.900331 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:25.902718 (+ 2387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:25.903628 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:25.903638 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:25.903767 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:58:25.904461 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:25.904480 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:25.905642 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:25.905647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:25.907185 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:25.907201 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:25.908173 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:25.908179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:25.908499 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:25.912370 (+ 3871us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:58:25.912373 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:58:25.912429 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:58:25.912559 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:58:25.912998 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:58:25.913005 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:58:25.913012 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 10:58:25.913212 (+ 200us) client_negotiation.cc:240] Negotiation successful
0806 10:58:25.913667 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":302,"threads_started":1}
I20260806 10:58:25.916337 17718 heartbeater.cc:347] Connected to a master server at 127.30.218.62:34189
I20260806 10:58:25.916684 17718 heartbeater.cc:464] Registering TS with master...
I20260806 10:58:25.917572 17718 heartbeater.cc:511] Master 127.30.218.62:34189 requested a full tablet report, sending...
I20260806 10:58:25.919238 17250 ts_manager.cc:194] Registered new tserver with Master: 6613c7d07fd041c9bb907353978e0592 (127.30.218.3:42247)
I20260806 10:58:25.920140 17250 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.218.3:41039
I20260806 10:58:25.922405 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:58:25.932926 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:25.926929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57256 (local address 127.30.218.62:34189)
0806 10:58:25.927176 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:25.927197 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:25.927538 (+ 341us) server_negotiation.cc:407] Connection header received
0806 10:58:25.927683 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:25.927687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:25.927765 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:25.927863 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:25.928379 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:25.929165 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:25.930677 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:25.931147 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:25.931263 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:25.932331 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:25.932336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:25.932340 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:25.932361 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:25.932397 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:25.932409 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:25.932484 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:25.932580 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 10:58:25.932670 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 10:58:25.937381 17250 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:58:25.945863 17250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:58:25.946061 17443 heartbeater.cc:503] Master 127.30.218.62:34189 was elected leader, sending a full tablet report...
I20260806 10:58:25.969650 17732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:25.958406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:44033 (local address 127.0.0.1:40530)
0806 10:58:25.960068 (+ 1662us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:25.960123 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:25.960307 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:25.961695 (+ 1388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:25.961704 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:25.961741 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:25.962227 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:25.962241 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:25.964687 (+ 2446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:25.964693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:25.966898 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:25.966912 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:25.968690 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:25.968696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:25.968908 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:25.968962 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 10:58:25.969064 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 10:58:25.969280 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1370,"thread_start_us":6958,"threads_started":1}
I20260806 10:58:25.971058 17731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:25.958617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40530 (local address 127.30.218.2:44033)
0806 10:58:25.960905 (+ 2288us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:25.960915 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:25.960949 (+ 34us) server_negotiation.cc:407] Connection header received
0806 10:58:25.961071 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:25.961080 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:25.961371 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:25.962512 (+ 1141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:25.962799 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:25.964500 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:25.967135 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:25.968465 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:25.970180 (+ 1715us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:25.970449 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:25.970535 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 10:58:25.970712 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2109,"thread_start_us":1954,"threads_started":1}
I20260806 10:58:25.975971 17733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:25.959772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.30.218.3:42247)
0806 10:58:25.965162 (+ 5390us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:25.965172 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:25.966069 (+ 897us) server_negotiation.cc:407] Connection header received
0806 10:58:25.966185 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:25.966193 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:25.966466 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:25.966668 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:25.967609 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:25.969083 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:25.972763 (+ 3680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:25.974198 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:25.974344 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:25.974599 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:25.975457 (+ 858us) server_negotiation.cc:299] Negotiation successful
0806 10:58:25.975640 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5175,"thread_start_us":4958,"threads_started":1}
I20260806 10:58:25.975310 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:25.959167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:42247 (local address 127.0.0.1:37666)
0806 10:58:25.965728 (+ 6561us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:25.965753 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:25.965888 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:25.967012 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:25.967018 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:25.967046 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:25.967416 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:25.967427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:25.969236 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:25.969241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:25.972525 (+ 3284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:25.972542 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:25.974422 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:25.974429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:25.974606 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:25.974640 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 10:58:25.974748 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 10:58:25.974889 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6418,"thread_start_us":755,"threads_started":1}
I20260806 10:58:25.975593 17512 tablet_service.cc:1467] Processing CreateTablet for tablet f2ca69a130434264a467340da08c8583 (DEFAULT_TABLE table=test-table [id=9b4ef06d299b4c73ad0cde2161574a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:58:25.977586 17512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2ca69a130434264a467340da08c8583. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:25.979176 17732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:25.967728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:38627 (local address 127.0.0.1:34462)
0806 10:58:25.969905 (+ 2177us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:25.969924 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:25.970020 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:25.972614 (+ 2594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:25.972619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:25.972646 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:25.972803 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:25.972814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:25.974910 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:25.974915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:25.976839 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:25.976855 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:25.978403 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:25.978409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:25.978602 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:25.978636 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 10:58:25.978725 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 10:58:25.978880 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2068,"thread_start_us":3644,"threads_started":1}
I20260806 10:58:25.979324 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:25.966280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.30.218.1:38627)
0806 10:58:25.971972 (+ 5692us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:25.971981 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:25.972013 (+ 32us) server_negotiation.cc:407] Connection header received
0806 10:58:25.972118 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:25.972127 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:25.972385 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:25.972998 (+ 613us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:25.973241 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:25.974726 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:25.977033 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:25.978221 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:25.978355 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:25.978630 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:25.978805 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 10:58:25.978969 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5502,"thread_start_us":6299,"threads_started":1}
I20260806 10:58:25.980450 17648 tablet_service.cc:1467] Processing CreateTablet for tablet f2ca69a130434264a467340da08c8583 (DEFAULT_TABLE table=test-table [id=9b4ef06d299b4c73ad0cde2161574a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:58:25.982051 17648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2ca69a130434264a467340da08c8583. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:25.982857 17374 tablet_service.cc:1467] Processing CreateTablet for tablet f2ca69a130434264a467340da08c8583 (DEFAULT_TABLE table=test-table [id=9b4ef06d299b4c73ad0cde2161574a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:58:25.984076 17374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2ca69a130434264a467340da08c8583. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:25.993512 17736 tablet_bootstrap.cc:492] T f2ca69a130434264a467340da08c8583 P 979ad97f5aab48bf8d83c7713711e720: Bootstrap starting.
I20260806 10:58:25.996512 17737 tablet_bootstrap.cc:492] T f2ca69a130434264a467340da08c8583 P 6613c7d07fd041c9bb907353978e0592: Bootstrap starting.
I20260806 10:58:25.997702 17736 tablet_bootstrap.cc:654] T f2ca69a130434264a467340da08c8583 P 979ad97f5aab48bf8d83c7713711e720: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:25.998121 17738 tablet_bootstrap.cc:492] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358: Bootstrap starting.
I20260806 10:58:25.999107 17736 log.cc:824] T f2ca69a130434264a467340da08c8583 P 979ad97f5aab48bf8d83c7713711e720: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:26.000088 17737 tablet_bootstrap.cc:654] T f2ca69a130434264a467340da08c8583 P 6613c7d07fd041c9bb907353978e0592: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:26.001304 17737 log.cc:824] T f2ca69a130434264a467340da08c8583 P 6613c7d07fd041c9bb907353978e0592: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:26.001549 17738 tablet_bootstrap.cc:654] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:26.002766 17738 log.cc:824] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:26.007977 17736 tablet_bootstrap.cc:492] T f2ca69a130434264a467340da08c8583 P 979ad97f5aab48bf8d83c7713711e720: No bootstrap required, opened a new log
I20260806 10:58:26.008236 17736 ts_tablet_manager.cc:1397] T f2ca69a130434264a467340da08c8583 P 979ad97f5aab48bf8d83c7713711e720: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260806 10:58:26.008255 17738 tablet_bootstrap.cc:492] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358: No bootstrap required, opened a new log
I20260806 10:58:26.008371 17737 tablet_bootstrap.cc:492] T f2ca69a130434264a467340da08c8583 P 6613c7d07fd041c9bb907353978e0592: No bootstrap required, opened a new log
I20260806 10:58:26.008503 17738 ts_tablet_manager.cc:1397] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260806 10:58:26.008613 17737 ts_tablet_manager.cc:1397] T f2ca69a130434264a467340da08c8583 P 6613c7d07fd041c9bb907353978e0592: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20260806 10:58:26.015908 17736 raft_consensus.cc:348] T f2ca69a130434264a467340da08c8583 P 979ad97f5aab48bf8d83c7713711e720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979ad97f5aab48bf8d83c7713711e720" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 44033 } } peers { permanent_uuid: "445f77820d504541b0e56297ac3a7358" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38627 } } peers { permanent_uuid: "6613c7d07fd041c9bb907353978e0592" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 42247 } }
I20260806 10:58:26.016326 17736 raft_consensus.cc:374] T f2ca69a130434264a467340da08c8583 P 979ad97f5aab48bf8d83c7713711e720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:26.016419 17736 raft_consensus.cc:729] T f2ca69a130434264a467340da08c8583 P 979ad97f5aab48bf8d83c7713711e720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 979ad97f5aab48bf8d83c7713711e720, State: Initialized, Role: FOLLOWER
I20260806 10:58:26.017081 17736 consensus_queue.cc:260] T f2ca69a130434264a467340da08c8583 P 979ad97f5aab48bf8d83c7713711e720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979ad97f5aab48bf8d83c7713711e720" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 44033 } } peers { permanent_uuid: "445f77820d504541b0e56297ac3a7358" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38627 } } peers { permanent_uuid: "6613c7d07fd041c9bb907353978e0592" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 42247 } }
I20260806 10:58:26.018260 17582 heartbeater.cc:503] Master 127.30.218.62:34189 was elected leader, sending a full tablet report...
I20260806 10:58:26.019033 17736 ts_tablet_manager.cc:1428] T f2ca69a130434264a467340da08c8583 P 979ad97f5aab48bf8d83c7713711e720: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260806 10:58:26.018985 17737 raft_consensus.cc:348] T f2ca69a130434264a467340da08c8583 P 6613c7d07fd041c9bb907353978e0592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979ad97f5aab48bf8d83c7713711e720" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 44033 } } peers { permanent_uuid: "445f77820d504541b0e56297ac3a7358" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38627 } } peers { permanent_uuid: "6613c7d07fd041c9bb907353978e0592" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 42247 } }
I20260806 10:58:26.018989 17738 raft_consensus.cc:348] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979ad97f5aab48bf8d83c7713711e720" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 44033 } } peers { permanent_uuid: "445f77820d504541b0e56297ac3a7358" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38627 } } peers { permanent_uuid: "6613c7d07fd041c9bb907353978e0592" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 42247 } }
I20260806 10:58:26.019486 17737 raft_consensus.cc:374] T f2ca69a130434264a467340da08c8583 P 6613c7d07fd041c9bb907353978e0592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:26.019486 17738 raft_consensus.cc:374] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:26.019600 17738 raft_consensus.cc:729] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445f77820d504541b0e56297ac3a7358, State: Initialized, Role: FOLLOWER
I20260806 10:58:26.019600 17737 raft_consensus.cc:729] T f2ca69a130434264a467340da08c8583 P 6613c7d07fd041c9bb907353978e0592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6613c7d07fd041c9bb907353978e0592, State: Initialized, Role: FOLLOWER
I20260806 10:58:26.020279 17738 consensus_queue.cc:260] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979ad97f5aab48bf8d83c7713711e720" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 44033 } } peers { permanent_uuid: "445f77820d504541b0e56297ac3a7358" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38627 } } peers { permanent_uuid: "6613c7d07fd041c9bb907353978e0592" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 42247 } }
I20260806 10:58:26.020279 17737 consensus_queue.cc:260] T f2ca69a130434264a467340da08c8583 P 6613c7d07fd041c9bb907353978e0592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979ad97f5aab48bf8d83c7713711e720" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 44033 } } peers { permanent_uuid: "445f77820d504541b0e56297ac3a7358" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38627 } } peers { permanent_uuid: "6613c7d07fd041c9bb907353978e0592" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 42247 } }
I20260806 10:58:26.022272 17737 ts_tablet_manager.cc:1428] T f2ca69a130434264a467340da08c8583 P 6613c7d07fd041c9bb907353978e0592: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260806 10:58:26.022606 17738 ts_tablet_manager.cc:1428] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260806 10:58:26.023901 17718 heartbeater.cc:503] Master 127.30.218.62:34189 was elected leader, sending a full tablet report...
W20260806 10:58:26.147769 17719 tablet.cc:2369] T f2ca69a130434264a467340da08c8583 P 6613c7d07fd041c9bb907353978e0592: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:58:26.160365 17583 tablet.cc:2369] T f2ca69a130434264a467340da08c8583 P 979ad97f5aab48bf8d83c7713711e720: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:58:26.174098 17444 tablet.cc:2369] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:58:26.315244 17744 raft_consensus.cc:482] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:58:26.315584 17744 raft_consensus.cc:504] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979ad97f5aab48bf8d83c7713711e720" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 44033 } } peers { permanent_uuid: "445f77820d504541b0e56297ac3a7358" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38627 } } peers { permanent_uuid: "6613c7d07fd041c9bb907353978e0592" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 42247 } }
I20260806 10:58:26.317484 17744 leader_election.cc:288] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 979ad97f5aab48bf8d83c7713711e720 (127.30.218.2:44033), 6613c7d07fd041c9bb907353978e0592 (127.30.218.3:42247)
I20260806 10:58:26.324797 17733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:26.318163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:54071 (local address 127.30.218.3:42247)
0806 10:58:26.318382 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:26.318389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:26.319047 (+ 658us) server_negotiation.cc:407] Connection header received
0806 10:58:26.319215 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:26.319220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:26.319304 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:26.319401 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:26.320007 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:26.320727 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:26.322904 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:26.323727 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:26.323885 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:26.323939 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:26.324344 (+ 405us) server_negotiation.cc:299] Negotiation successful
0806 10:58:26.324537 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 10:58:26.324838 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:26.317859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:42247 (local address 127.30.218.1:54071)
0806 10:58:26.318900 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:26.318922 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:26.319067 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:26.319432 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:26.319435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:26.319470 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:26.319836 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:26.319845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:26.320867 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:26.320871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:26.322692 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:26.322707 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:26.323926 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:26.323933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:26.324141 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:26.324208 (+ 67us) client_negotiation.cc:769] Sending connection context
0806 10:58:26.324292 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 10:58:26.324405 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"mutex_wait_us":79,"spinlock_wait_cycles":1920,"thread_start_us":488,"threads_started":1}
I20260806 10:58:26.324882 17731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:26.317865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:33277 (local address 127.30.218.2:44033)
0806 10:58:26.318100 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:26.318108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:26.318879 (+ 771us) server_negotiation.cc:407] Connection header received
0806 10:58:26.319053 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:26.319059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:26.319162 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:26.319241 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:26.319947 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:26.320683 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:26.322957 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:26.323743 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:26.323907 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:26.323980 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:26.324482 (+ 502us) server_negotiation.cc:299] Negotiation successful
0806 10:58:26.324637 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 10:58:26.324929 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:26.317741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:44033 (local address 127.30.218.1:33277)
0806 10:58:26.318741 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:26.318762 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:26.318907 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:26.319290 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:26.319295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:26.319321 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:26.319774 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:26.319788 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:26.320831 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:26.320836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:26.322697 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:26.322710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:26.324098 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:26.324103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:26.324306 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:26.324339 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 10:58:26.324426 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 10:58:26.324553 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":132,"thread_start_us":861,"threads_started":1}
I20260806 10:58:26.326217 17669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ca69a130434264a467340da08c8583" candidate_uuid: "445f77820d504541b0e56297ac3a7358" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6613c7d07fd041c9bb907353978e0592" is_pre_election: true
I20260806 10:58:26.326210 17533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ca69a130434264a467340da08c8583" candidate_uuid: "445f77820d504541b0e56297ac3a7358" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "979ad97f5aab48bf8d83c7713711e720" is_pre_election: true
I20260806 10:58:26.326859 17669 raft_consensus.cc:2454] T f2ca69a130434264a467340da08c8583 P 6613c7d07fd041c9bb907353978e0592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 445f77820d504541b0e56297ac3a7358 in term 0.
I20260806 10:58:26.326859 17533 raft_consensus.cc:2454] T f2ca69a130434264a467340da08c8583 P 979ad97f5aab48bf8d83c7713711e720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 445f77820d504541b0e56297ac3a7358 in term 0.
I20260806 10:58:26.327661 17328 leader_election.cc:302] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 445f77820d504541b0e56297ac3a7358, 6613c7d07fd041c9bb907353978e0592, 979ad97f5aab48bf8d83c7713711e720; no voters:
I20260806 10:58:26.328208 17744 raft_consensus.cc:2781] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:58:26.328399 17744 raft_consensus.cc:482] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:58:26.328500 17744 raft_consensus.cc:3037] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:26.331032 17744 raft_consensus.cc:504] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979ad97f5aab48bf8d83c7713711e720" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 44033 } } peers { permanent_uuid: "445f77820d504541b0e56297ac3a7358" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38627 } } peers { permanent_uuid: "6613c7d07fd041c9bb907353978e0592" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 42247 } }
I20260806 10:58:26.331801 17744 leader_election.cc:288] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358 [CANDIDATE]: Term 1 election: Requested vote from peers 979ad97f5aab48bf8d83c7713711e720 (127.30.218.2:44033), 6613c7d07fd041c9bb907353978e0592 (127.30.218.3:42247)
I20260806 10:58:26.332414 17533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ca69a130434264a467340da08c8583" candidate_uuid: "445f77820d504541b0e56297ac3a7358" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "979ad97f5aab48bf8d83c7713711e720"
I20260806 10:58:26.332491 17669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ca69a130434264a467340da08c8583" candidate_uuid: "445f77820d504541b0e56297ac3a7358" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6613c7d07fd041c9bb907353978e0592"
I20260806 10:58:26.332708 17669 raft_consensus.cc:3037] T f2ca69a130434264a467340da08c8583 P 6613c7d07fd041c9bb907353978e0592 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:26.332710 17533 raft_consensus.cc:3037] T f2ca69a130434264a467340da08c8583 P 979ad97f5aab48bf8d83c7713711e720 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:26.336033 17669 raft_consensus.cc:2454] T f2ca69a130434264a467340da08c8583 P 6613c7d07fd041c9bb907353978e0592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 445f77820d504541b0e56297ac3a7358 in term 1.
I20260806 10:58:26.336036 17533 raft_consensus.cc:2454] T f2ca69a130434264a467340da08c8583 P 979ad97f5aab48bf8d83c7713711e720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 445f77820d504541b0e56297ac3a7358 in term 1.
I20260806 10:58:26.336637 17328 leader_election.cc:302] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 445f77820d504541b0e56297ac3a7358, 6613c7d07fd041c9bb907353978e0592, 979ad97f5aab48bf8d83c7713711e720; no voters:
I20260806 10:58:26.336983 17744 raft_consensus.cc:2781] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:58:26.337425 17744 raft_consensus.cc:686] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358 [term 1 LEADER]: Becoming Leader. State: Replica: 445f77820d504541b0e56297ac3a7358, State: Running, Role: LEADER
I20260806 10:58:26.338027 17744 consensus_queue.cc:237] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979ad97f5aab48bf8d83c7713711e720" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 44033 } } peers { permanent_uuid: "445f77820d504541b0e56297ac3a7358" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38627 } } peers { permanent_uuid: "6613c7d07fd041c9bb907353978e0592" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 42247 } }
I20260806 10:58:26.342828 17249 catalog_manager.cc:5495] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358 reported cstate change: term changed from 0 to 1, leader changed from <none> to 445f77820d504541b0e56297ac3a7358 (127.30.218.1). New cstate: current_term: 1 leader_uuid: "445f77820d504541b0e56297ac3a7358" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979ad97f5aab48bf8d83c7713711e720" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "445f77820d504541b0e56297ac3a7358" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 38627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6613c7d07fd041c9bb907353978e0592" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 42247 } health_report { overall_health: UNKNOWN } } }
I20260806 10:58:26.429246 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:26.424029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34472 (local address 127.30.218.1:38627)
0806 10:58:26.424195 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:26.424199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:26.424217 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:58:26.424349 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:26.424353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:26.424459 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:26.424587 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:58:26.424971 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:26.425802 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:26.427254 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:26.427681 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:26.427772 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:26.428225 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:58:26.428750 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:58:26.428856 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:26.428959 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 10:58:26.429031 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 10:58:26.438107 17669 raft_consensus.cc:1261] T f2ca69a130434264a467340da08c8583 P 6613c7d07fd041c9bb907353978e0592 [term 1 FOLLOWER]: Refusing update from remote peer 445f77820d504541b0e56297ac3a7358: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:58:26.438117 17533 raft_consensus.cc:1261] T f2ca69a130434264a467340da08c8583 P 979ad97f5aab48bf8d83c7713711e720 [term 1 FOLLOWER]: Refusing update from remote peer 445f77820d504541b0e56297ac3a7358: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:58:26.439218 17747 consensus_queue.cc:1035] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358 [LEADER]: Connected to new peer: Peer: permanent_uuid: "979ad97f5aab48bf8d83c7713711e720" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:58:26.439620 17744 consensus_queue.cc:1035] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6613c7d07fd041c9bb907353978e0592" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 42247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:58:26.465152 17250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:58:26.465584 17250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:58:26.469555 17250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e518b2f190a40a2a39787787800bccc: Sending DeleteTablet for 3 replicas of tablet f2ca69a130434264a467340da08c8583
I20260806 10:58:26.471038 17512 tablet_service.cc:1514] Processing DeleteTablet for tablet f2ca69a130434264a467340da08c8583 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:58:26 UTC) from {username='slave'} at 127.0.0.1:40530
I20260806 10:58:26.471038 17648 tablet_service.cc:1514] Processing DeleteTablet for tablet f2ca69a130434264a467340da08c8583 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:58:26 UTC) from {username='slave'} at 127.0.0.1:37666
I20260806 10:58:26.471460 17374 tablet_service.cc:1514] Processing DeleteTablet for tablet f2ca69a130434264a467340da08c8583 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:58:26 UTC) from {username='slave'} at 127.0.0.1:34462
I20260806 10:58:26.473292 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 17311
I20260806 10:58:26.482079 17762 tablet_replica.cc:330] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358: stopping tablet replica
I20260806 10:58:26.482599 17762 raft_consensus.cc:2229] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:58:26.483333 17762 raft_consensus.cc:2258] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:58:26.485670 17762 ts_tablet_manager.cc:1905] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:58:26.493499 17762 ts_tablet_manager.cc:1918] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:58:26.493690 17762 log.cc:1196] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-0/wal/wals/f2ca69a130434264a467340da08c8583
I20260806 10:58:26.494249 17762 ts_tablet_manager.cc:1939] T f2ca69a130434264a467340da08c8583 P 445f77820d504541b0e56297ac3a7358: Deleting consensus metadata
I20260806 10:58:26.496203 17233 catalog_manager.cc:4841] TS 445f77820d504541b0e56297ac3a7358 (127.30.218.1:38627): tablet f2ca69a130434264a467340da08c8583 (table test-table [id=9b4ef06d299b4c73ad0cde2161574a4c]) successfully deleted
I20260806 10:58:26.500415 17760 tablet_replica.cc:330] T f2ca69a130434264a467340da08c8583 P 6613c7d07fd041c9bb907353978e0592: stopping tablet replica
I20260806 10:58:26.500810 17761 tablet_replica.cc:330] T f2ca69a130434264a467340da08c8583 P 979ad97f5aab48bf8d83c7713711e720: stopping tablet replica
I20260806 10:58:26.500936 17760 raft_consensus.cc:2229] T f2ca69a130434264a467340da08c8583 P 6613c7d07fd041c9bb907353978e0592 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:58:26.501350 17761 raft_consensus.cc:2229] T f2ca69a130434264a467340da08c8583 P 979ad97f5aab48bf8d83c7713711e720 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:58:26.501529 17760 raft_consensus.cc:2258] T f2ca69a130434264a467340da08c8583 P 6613c7d07fd041c9bb907353978e0592 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:58:26.502004 17761 raft_consensus.cc:2258] T f2ca69a130434264a467340da08c8583 P 979ad97f5aab48bf8d83c7713711e720 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:58:26.504354 17761 ts_tablet_manager.cc:1905] T f2ca69a130434264a467340da08c8583 P 979ad97f5aab48bf8d83c7713711e720: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:58:26.504354 17760 ts_tablet_manager.cc:1905] T f2ca69a130434264a467340da08c8583 P 6613c7d07fd041c9bb907353978e0592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:58:26.508505 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:26.476843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34488 (local address 127.30.218.1:38627)
0806 10:58:26.477030 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:26.477037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:26.499661 (+ 22624us) server_negotiation.cc:407] Connection header received
0806 10:58:26.499869 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:26.499874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:26.499984 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:26.500068 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:26.500768 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:26.501938 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:26.504371 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:26.505003 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:26.505114 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:26.506715 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:26.506727 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:26.506744 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:26.507556 (+ 812us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:26.507632 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:26.507660 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:26.507742 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:26.507996 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 10:58:26.508120 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 10:58:26.509666 17438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:26.511293 17760 ts_tablet_manager.cc:1918] T f2ca69a130434264a467340da08c8583 P 6613c7d07fd041c9bb907353978e0592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:58:26.511466 17760 log.cc:1196] T f2ca69a130434264a467340da08c8583 P 6613c7d07fd041c9bb907353978e0592: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/f2ca69a130434264a467340da08c8583
I20260806 10:58:26.511972 17760 ts_tablet_manager.cc:1939] T f2ca69a130434264a467340da08c8583 P 6613c7d07fd041c9bb907353978e0592: Deleting consensus metadata
I20260806 10:58:26.512634 17761 ts_tablet_manager.cc:1918] T f2ca69a130434264a467340da08c8583 P 979ad97f5aab48bf8d83c7713711e720: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:58:26.512845 17761 log.cc:1196] T f2ca69a130434264a467340da08c8583 P 979ad97f5aab48bf8d83c7713711e720: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/f2ca69a130434264a467340da08c8583
I20260806 10:58:26.513413 17761 ts_tablet_manager.cc:1939] T f2ca69a130434264a467340da08c8583 P 979ad97f5aab48bf8d83c7713711e720: Deleting consensus metadata
I20260806 10:58:26.514623 17234 catalog_manager.cc:4841] TS 6613c7d07fd041c9bb907353978e0592 (127.30.218.3:42247): tablet f2ca69a130434264a467340da08c8583 (table test-table [id=9b4ef06d299b4c73ad0cde2161574a4c]) successfully deleted
I20260806 10:58:26.515278 17233 catalog_manager.cc:4841] TS 979ad97f5aab48bf8d83c7713711e720 (127.30.218.2:44033): tablet f2ca69a130434264a467340da08c8583 (table test-table [id=9b4ef06d299b4c73ad0cde2161574a4c]) successfully deleted
I20260806 10:58:26.612557 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 17311
I20260806 10:58:26.632875 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 17447
I20260806 10:58:26.640410 17731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:26.633783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40546 (local address 127.30.218.2:44033)
0806 10:58:26.633952 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:26.633957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:26.633978 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:58:26.634154 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:26.634158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:26.634252 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:26.634347 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:26.634775 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:26.635565 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:26.636987 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:26.637420 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:26.637587 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:26.638889 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:26.638899 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:26.638916 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:26.639577 (+ 661us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:26.639633 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:26.639655 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:26.639761 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:26.639923 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 10:58:26.640055 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 10:58:26.641146 17577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:26.731243 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 17447
I20260806 10:58:26.748399 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 17585
I20260806 10:58:26.755126 17733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:26.749153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37670 (local address 127.30.218.3:42247)
0806 10:58:26.749339 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:26.749343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:26.749362 (+ 19us) server_negotiation.cc:407] Connection header received
0806 10:58:26.749575 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:26.749579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:26.749697 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:26.749790 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:26.750154 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:26.750938 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:26.752272 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:26.752706 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:26.752810 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:26.753873 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:26.753882 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:26.753897 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:26.754480 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:26.754532 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:26.754553 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:26.754670 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:26.754754 (+ 84us) server_negotiation.cc:299] Negotiation successful
0806 10:58:26.754858 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 10:58:26.755803 17712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:26.846310 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 17585
I20260806 10:58:26.863296 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 17217
I20260806 10:58:26.864522 17282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:26.976680 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 17217
2026-08-06T10:58:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3246 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:58:27 dist-test-slave-49h3 krb5kdc[17771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:58:27 dist-test-slave-49h3 krb5kdc[17771](info): set up 2 sockets
Aug 06 10:58:27 dist-test-slave-49h3 krb5kdc[17771](info): commencing operation
krb5kdc: starting...
W20260806 10:58:28.054001 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.004s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:58:28 dist-test-slave-49h3 krb5kdc[17771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:58:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:58:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:58:28.389294 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:41633
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:38089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:41633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:58:28.742985 17784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:28.743320 17784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:28.743455 17784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:28.751092 17784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:58:28.751216 17784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:28.751286 17784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:28.751369 17784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:58:28.751426 17784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:58:28.761976 17784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:38089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:41633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:41633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:28.763418 17784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:28.764780 17784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:28.777936 17789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:28.777956 17790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:28.777875 17792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:28.778587 17784 server_base.cc:1034] running on GCE node
I20260806 10:58:28.779395 17784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:28.780884 17784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:28.782087 17784 hybrid_clock.cc:648] HybridClock initialized: now 1786013908782027 us; error 56 us; skew 500 ppm
I20260806 10:58:28.782436 17784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:58:28 dist-test-slave-49h3 krb5kdc[17771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013908, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:58:28.785682 17784 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:58:28.787331 17784 webserver.cc:460] Webserver started at http://127.30.218.62:46067/ using document root <none> and password file <none>
I20260806 10:58:28.788075 17784 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:28.788209 17784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:28.788527 17784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:28.791893 17784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "5a87b13e952e44b7abd96bf791c2e66b"
format_stamp: "Formatted at 2026-08-06 10:58:28 on dist-test-slave-49h3"
server_key: "5fa6ea8ffd897973538b90f29075b0cf"
server_key_iv: "db62662e01a3bcdbff8c8b42dfb1d2d4"
server_key_version: "encryptionkey@0"
I20260806 10:58:28.792740 17784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "5a87b13e952e44b7abd96bf791c2e66b"
format_stamp: "Formatted at 2026-08-06 10:58:28 on dist-test-slave-49h3"
server_key: "5fa6ea8ffd897973538b90f29075b0cf"
server_key_iv: "db62662e01a3bcdbff8c8b42dfb1d2d4"
server_key_version: "encryptionkey@0"
I20260806 10:58:28.798579 17784 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 10:58:28.802523 17799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:28.803979 17784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:58:28.804176 17784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "5a87b13e952e44b7abd96bf791c2e66b"
format_stamp: "Formatted at 2026-08-06 10:58:28 on dist-test-slave-49h3"
server_key: "5fa6ea8ffd897973538b90f29075b0cf"
server_key_iv: "db62662e01a3bcdbff8c8b42dfb1d2d4"
server_key_version: "encryptionkey@0"
I20260806 10:58:28.804405 17784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:28.823982 17784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:28.828575 17784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:28.828889 17784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:28.847550 17784 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:41633
I20260806 10:58:28.847572 17854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:41633 every 8 connection(s)
I20260806 10:58:28.849000 17784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:58:28.852983 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 17784
I20260806 10:58:28.853144 17855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:28.853184 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:58:28.853703 31592 external_mini_cluster.cc:1428] Setting key 758cc0a5d7a3535979a1bad8ba5f9ae5
I20260806 10:58:28.869112 17855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a87b13e952e44b7abd96bf791c2e66b: Bootstrap starting.
Aug 06 10:58:28 dist-test-slave-49h3 krb5kdc[17771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:58:28.872957 17855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a87b13e952e44b7abd96bf791c2e66b: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:28.874285 17855 log.cc:824] T 00000000000000000000000000000000 P 5a87b13e952e44b7abd96bf791c2e66b: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:28.877483 17855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a87b13e952e44b7abd96bf791c2e66b: No bootstrap required, opened a new log
I20260806 10:58:28.879158 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:28.855590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48074 (local address 127.30.218.62:41633)
0806 10:58:28.856477 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:28.856490 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:28.856540 (+ 50us) server_negotiation.cc:407] Connection header received
0806 10:58:28.858307 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:28.858336 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:28.858825 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:28.859227 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:28.861244 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:28.862663 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:28.864638 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:28.865642 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:28.865806 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:28.871091 (+ 5285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:28.871101 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:28.871119 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:28.871173 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:28.874615 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:28.875515 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:28.875524 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:28.875531 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:28.875724 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:28.876149 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:28.876153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:28.876156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:28.876768 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:28.877066 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:28.877597 (+ 531us) server_negotiation.cc:299] Negotiation successful
0806 10:58:28.878097 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":381,"threads_started":1}
I20260806 10:58:28.886361 17855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a87b13e952e44b7abd96bf791c2e66b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a87b13e952e44b7abd96bf791c2e66b" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41633 } }
I20260806 10:58:28.886816 17855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a87b13e952e44b7abd96bf791c2e66b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:28.886941 17855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a87b13e952e44b7abd96bf791c2e66b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a87b13e952e44b7abd96bf791c2e66b, State: Initialized, Role: FOLLOWER
I20260806 10:58:28.887641 17855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a87b13e952e44b7abd96bf791c2e66b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a87b13e952e44b7abd96bf791c2e66b" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41633 } }
I20260806 10:58:28.887959 17855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a87b13e952e44b7abd96bf791c2e66b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:58:28.888128 17855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a87b13e952e44b7abd96bf791c2e66b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:58:28.888338 17855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a87b13e952e44b7abd96bf791c2e66b [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:28.890719 17855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a87b13e952e44b7abd96bf791c2e66b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a87b13e952e44b7abd96bf791c2e66b" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41633 } }
I20260806 10:58:28.891237 17855 leader_election.cc:302] T 00000000000000000000000000000000 P 5a87b13e952e44b7abd96bf791c2e66b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a87b13e952e44b7abd96bf791c2e66b; no voters:
I20260806 10:58:28.891785 17855 leader_election.cc:288] T 00000000000000000000000000000000 P 5a87b13e952e44b7abd96bf791c2e66b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:58:28.891943 17860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a87b13e952e44b7abd96bf791c2e66b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:58:28.892635 17860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a87b13e952e44b7abd96bf791c2e66b [term 1 LEADER]: Becoming Leader. State: Replica: 5a87b13e952e44b7abd96bf791c2e66b, State: Running, Role: LEADER
I20260806 10:58:28.893224 17860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a87b13e952e44b7abd96bf791c2e66b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a87b13e952e44b7abd96bf791c2e66b" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41633 } }
I20260806 10:58:28.894017 17855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a87b13e952e44b7abd96bf791c2e66b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:58:28.896288 17861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a87b13e952e44b7abd96bf791c2e66b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a87b13e952e44b7abd96bf791c2e66b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a87b13e952e44b7abd96bf791c2e66b" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41633 } } }
I20260806 10:58:28.896178 17862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a87b13e952e44b7abd96bf791c2e66b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a87b13e952e44b7abd96bf791c2e66b. Latest consensus state: current_term: 1 leader_uuid: "5a87b13e952e44b7abd96bf791c2e66b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a87b13e952e44b7abd96bf791c2e66b" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 41633 } } }
I20260806 10:58:28.896552 17861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a87b13e952e44b7abd96bf791c2e66b [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:28.896552 17862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a87b13e952e44b7abd96bf791c2e66b [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:28.898512 17867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:58:28.903584 17867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:58:28.912985 17867 catalog_manager.cc:1347] Generated new cluster ID: 45d602ca2c65491085c67c3ebe222bd9
I20260806 10:58:28.913102 17867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:58:28.932096 17867 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:58:28.934124 17867 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:58:28.947638 17867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a87b13e952e44b7abd96bf791c2e66b: Generated new TSK 0
I20260806 10:58:28.948782 17867 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
I20260806 10:58:29.041361 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:41633
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:38089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:58:29.394433 17882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:29.394800 17882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:29.394943 17882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:29.403139 17882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:29.403304 17882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:29.403475 17882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:58:29.414390 17882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:38089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:41633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:29.415797 17882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:29.417234 17882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:29.431242 17887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:29.431324 17888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:29.431233 17890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:29.431968 17882 server_base.cc:1034] running on GCE node
I20260806 10:58:29.432685 17882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:29.433877 17882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:29.435109 17882 hybrid_clock.cc:648] HybridClock initialized: now 1786013909435009 us; error 118 us; skew 500 ppm
I20260806 10:58:29.435446 17882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:58:29 dist-test-slave-49h3 krb5kdc[17771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013909, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:58:29.438902 17882 init.cc:377] Logged in from keytab as kudu/127.30.218.1@KRBTEST.COM (short username kudu)
I20260806 10:58:29.440476 17882 webserver.cc:460] Webserver started at http://127.30.218.1:45669/ using document root <none> and password file <none>
I20260806 10:58:29.441177 17882 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:29.441290 17882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:29.441632 17882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:29.444617 17882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "19f43c6d4d6b49f7840945eed04e37d8"
format_stamp: "Formatted at 2026-08-06 10:58:29 on dist-test-slave-49h3"
server_key: "6ea9f6a1163711cc13e5cd7d83b4afe3"
server_key_iv: "a6dbd5655e9ac44ef817d8c085911186"
server_key_version: "encryptionkey@0"
I20260806 10:58:29.445353 17882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "19f43c6d4d6b49f7840945eed04e37d8"
format_stamp: "Formatted at 2026-08-06 10:58:29 on dist-test-slave-49h3"
server_key: "6ea9f6a1163711cc13e5cd7d83b4afe3"
server_key_iv: "a6dbd5655e9ac44ef817d8c085911186"
server_key_version: "encryptionkey@0"
I20260806 10:58:29.450734 17882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 10:58:29.454227 17897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:29.455670 17882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:58:29.455821 17882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "19f43c6d4d6b49f7840945eed04e37d8"
format_stamp: "Formatted at 2026-08-06 10:58:29 on dist-test-slave-49h3"
server_key: "6ea9f6a1163711cc13e5cd7d83b4afe3"
server_key_iv: "a6dbd5655e9ac44ef817d8c085911186"
server_key_version: "encryptionkey@0"
I20260806 10:58:29.456009 17882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:29.478920 17882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:29.483382 17882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:29.483675 17882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:29.484747 17882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:58:29.486217 17882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:58:29.486299 17882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:29.486397 17882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:58:29.486445 17882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:29.519306 17882 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:43297
I20260806 10:58:29.519344 18011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:43297 every 8 connection(s)
I20260806 10:58:29.521010 17882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:58:29.526522 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 17882
I20260806 10:58:29.526698 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:58:29.527158 31592 external_mini_cluster.cc:1428] Setting key 4483dc8b3c1d3be639cfe757a99e85c9
Aug 06 10:58:29 dist-test-slave-49h3 krb5kdc[17771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013909, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:58:29.541728 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:29.523719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:38069 (local address 127.30.218.62:41633)
0806 10:58:29.524176 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:29.524181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:29.525253 (+ 1072us) server_negotiation.cc:407] Connection header received
0806 10:58:29.527304 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:29.527310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:29.527401 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:29.527502 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:29.529273 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:29.530115 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:29.531765 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:29.532195 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:29.532320 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:29.536050 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:29.536055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:29.536059 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:29.536094 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:29.538600 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:29.539357 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:29.539362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:29.539364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:29.539492 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:29.539917 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:29.539921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:29.539923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:29.540192 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:29.540337 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:29.541077 (+ 740us) server_negotiation.cc:299] Negotiation successful
0806 10:58:29.541325 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":224,"threads_started":1}
I20260806 10:58:29.542723 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:29.524279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:41633 (local address 127.30.218.1:38069)
0806 10:58:29.525042 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:29.525114 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:29.526984 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:29.527776 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:29.527785 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:29.528373 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:58:29.528991 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:29.529010 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:29.530294 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:29.530298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:29.531610 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:29.531625 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:29.532452 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:29.532460 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:29.532698 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:29.533434 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:58:29.533496 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:58:29.535798 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:58:29.538749 (+ 2951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:58:29.538760 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:58:29.538781 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:58:29.539236 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:58:29.539600 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:58:29.539605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:58:29.539609 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:58:29.539810 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:58:29.540333 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:58:29.540342 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:58:29.540697 (+ 355us) client_negotiation.cc:769] Sending connection context
0806 10:58:29.540953 (+ 256us) client_negotiation.cc:240] Negotiation successful
0806 10:58:29.541554 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":284,"threads_started":1}
I20260806 10:58:29.544286 18012 heartbeater.cc:347] Connected to a master server at 127.30.218.62:41633
I20260806 10:58:29.544571 18012 heartbeater.cc:464] Registering TS with master...
I20260806 10:58:29.545279 18012 heartbeater.cc:511] Master 127.30.218.62:41633 requested a full tablet report, sending...
I20260806 10:58:29.547381 17816 ts_manager.cc:194] Registered new tserver with Master: 19f43c6d4d6b49f7840945eed04e37d8 (127.30.218.1:43297)
I20260806 10:58:29.548800 17816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.1@KRBTEST.COM'} at 127.30.218.1:38069
WARNING: no policy specified for kudu/127.30.218.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
I20260806 10:58:29.596611 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:41633
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:38089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:58:29.944902 18018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:29.945214 18018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:29.945314 18018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:29.953013 18018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:29.953157 18018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:29.953284 18018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:58:29.963851 18018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:38089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:41633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:29.965199 18018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:29.966588 18018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:29.979589 18023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:29.979628 18024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:29.980224 18027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:29.981527 18018 server_base.cc:1034] running on GCE node
I20260806 10:58:29.982191 18018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:29.983187 18018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:29.984385 18018 hybrid_clock.cc:648] HybridClock initialized: now 1786013909984336 us; error 41 us; skew 500 ppm
I20260806 10:58:29.984699 18018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:58:29 dist-test-slave-49h3 krb5kdc[17771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013909, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:58:29.987931 18018 init.cc:377] Logged in from keytab as kudu/127.30.218.2@KRBTEST.COM (short username kudu)
I20260806 10:58:29.989413 18018 webserver.cc:460] Webserver started at http://127.30.218.2:38135/ using document root <none> and password file <none>
I20260806 10:58:29.990201 18018 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:29.990315 18018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:29.990630 18018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:29.993652 18018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "37a5cb5a1aa7472a878cff751ca9f3d9"
format_stamp: "Formatted at 2026-08-06 10:58:29 on dist-test-slave-49h3"
server_key: "3021112ed9bcff25789c67fb7f701f58"
server_key_iv: "d62f3a4e3dc64501dc320524177bf1a0"
server_key_version: "encryptionkey@0"
I20260806 10:58:29.994460 18018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "37a5cb5a1aa7472a878cff751ca9f3d9"
format_stamp: "Formatted at 2026-08-06 10:58:29 on dist-test-slave-49h3"
server_key: "3021112ed9bcff25789c67fb7f701f58"
server_key_iv: "d62f3a4e3dc64501dc320524177bf1a0"
server_key_version: "encryptionkey@0"
I20260806 10:58:29.999890 18018 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:58:30.003468 18034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:30.004842 18018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:58:30.005010 18018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "37a5cb5a1aa7472a878cff751ca9f3d9"
format_stamp: "Formatted at 2026-08-06 10:58:29 on dist-test-slave-49h3"
server_key: "3021112ed9bcff25789c67fb7f701f58"
server_key_iv: "d62f3a4e3dc64501dc320524177bf1a0"
server_key_version: "encryptionkey@0"
I20260806 10:58:30.005208 18018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:30.035986 18018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:30.039996 18018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:30.040294 18018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:30.041273 18018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:58:30.042683 18018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:58:30.042765 18018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:30.042886 18018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:58:30.042949 18018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:30.075373 18018 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:38495
I20260806 10:58:30.075419 18150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:38495 every 8 connection(s)
I20260806 10:58:30.076768 18018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:58:30.081983 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 18018
I20260806 10:58:30.082197 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:58:30.082783 31592 external_mini_cluster.cc:1428] Setting key 1a0b3b04f396d50f52b64dd1555a3572
Aug 06 10:58:30 dist-test-slave-49h3 krb5kdc[17771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013909, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:58:30.101290 18153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:30.079490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:58763 (local address 127.30.218.62:41633)
0806 10:58:30.079946 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:30.079950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:30.080889 (+ 939us) server_negotiation.cc:407] Connection header received
0806 10:58:30.083569 (+ 2680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:30.083575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:30.083683 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:30.083808 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:30.085877 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.086647 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.089003 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.089546 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.089652 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:30.094289 (+ 4637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:30.094296 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:30.094299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:30.094341 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:30.097766 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:30.098743 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:30.098749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:30.098752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:30.098924 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:30.099430 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:30.099434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:30.099437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:30.099807 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:30.099956 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:30.100789 (+ 833us) server_negotiation.cc:299] Negotiation successful
0806 10:58:30.101006 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":241,"threads_started":1}
I20260806 10:58:30.102864 18154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:30.079956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:41633 (local address 127.30.218.2:58763)
0806 10:58:30.080707 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:30.080756 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:30.083120 (+ 2364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:30.084098 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:30.084109 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:30.084766 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:58:30.085669 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:30.085703 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.086821 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.086827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:30.088811 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:30.088830 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.089813 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.089819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:30.090121 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:30.090912 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:58:30.090942 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:58:30.093922 (+ 2980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:58:30.097979 (+ 4057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:58:30.097988 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:58:30.098006 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:58:30.098589 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:58:30.099047 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:58:30.099051 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:58:30.099055 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:58:30.099298 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:58:30.099951 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:58:30.099962 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:58:30.100385 (+ 423us) client_negotiation.cc:769] Sending connection context
0806 10:58:30.100648 (+ 263us) client_negotiation.cc:240] Negotiation successful
0806 10:58:30.101289 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":296,"threads_started":1}
I20260806 10:58:30.104578 18151 heartbeater.cc:347] Connected to a master server at 127.30.218.62:41633
I20260806 10:58:30.104900 18151 heartbeater.cc:464] Registering TS with master...
I20260806 10:58:30.105746 18151 heartbeater.cc:511] Master 127.30.218.62:41633 requested a full tablet report, sending...
I20260806 10:58:30.107568 17816 ts_manager.cc:194] Registered new tserver with Master: 37a5cb5a1aa7472a878cff751ca9f3d9 (127.30.218.2:38495)
I20260806 10:58:30.108693 17816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.2@KRBTEST.COM'} at 127.30.218.2:58763
WARNING: no policy specified for kudu/127.30.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
I20260806 10:58:30.157032 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:41633
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:38089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:58:30.496567 18157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:30.496892 18157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:30.497020 18157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:30.504554 18157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:30.504709 18157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:30.504863 18157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:58:30.515267 18157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:38089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:41633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:30.516695 18157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:30.518172 18157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:30.533944 18163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:30.534004 18162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:30.533975 18165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:30.535142 18157 server_base.cc:1034] running on GCE node
I20260806 10:58:30.535816 18157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:30.536821 18157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:30.538053 18157 hybrid_clock.cc:648] HybridClock initialized: now 1786013910537986 us; error 57 us; skew 500 ppm
I20260806 10:58:30.538429 18157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:58:30 dist-test-slave-49h3 krb5kdc[17771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013910, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:58:30.541838 18157 init.cc:377] Logged in from keytab as kudu/127.30.218.3@KRBTEST.COM (short username kudu)
I20260806 10:58:30.543530 18157 webserver.cc:460] Webserver started at http://127.30.218.3:42323/ using document root <none> and password file <none>
I20260806 10:58:30.544294 18157 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:30.544430 18157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:30.544831 18157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:30.548197 18157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "b363584ad3be4198953a42d8ef1d24ac"
format_stamp: "Formatted at 2026-08-06 10:58:30 on dist-test-slave-49h3"
server_key: "f0d180377036118f214c59253d52a7ec"
server_key_iv: "847d0863f0ec44699e6404f07ca0372c"
server_key_version: "encryptionkey@0"
I20260806 10:58:30.549063 18157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "b363584ad3be4198953a42d8ef1d24ac"
format_stamp: "Formatted at 2026-08-06 10:58:30 on dist-test-slave-49h3"
server_key: "f0d180377036118f214c59253d52a7ec"
server_key_iv: "847d0863f0ec44699e6404f07ca0372c"
server_key_version: "encryptionkey@0"
I20260806 10:58:30.551676 18012 heartbeater.cc:503] Master 127.30.218.62:41633 was elected leader, sending a full tablet report...
I20260806 10:58:30.554970 18157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260806 10:58:30.558845 18173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:30.560443 18157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:58:30.560629 18157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "b363584ad3be4198953a42d8ef1d24ac"
format_stamp: "Formatted at 2026-08-06 10:58:30 on dist-test-slave-49h3"
server_key: "f0d180377036118f214c59253d52a7ec"
server_key_iv: "847d0863f0ec44699e6404f07ca0372c"
server_key_version: "encryptionkey@0"
I20260806 10:58:30.560853 18157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:30.577260 18157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:30.581629 18157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:30.581900 18157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:30.582863 18157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:58:30.584182 18157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:58:30.584271 18157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:30.584395 18157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:58:30.584450 18157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:30.615698 18157 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:37161
I20260806 10:58:30.615739 18289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:37161 every 8 connection(s)
I20260806 10:58:30.617084 18157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:58:30.619817 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 18157
I20260806 10:58:30.620018 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:58:30.620529 31592 external_mini_cluster.cc:1428] Setting key dafbaa1d5a1c3ba50b66730f17788dc6
Aug 06 10:58:30 dist-test-slave-49h3 krb5kdc[17771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013910, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:58:30.638484 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:30.619819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:52219 (local address 127.30.218.62:41633)
0806 10:58:30.620371 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:30.620378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:30.621431 (+ 1053us) server_negotiation.cc:407] Connection header received
0806 10:58:30.623704 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:30.623709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:30.623784 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:30.623874 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:30.625508 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.626307 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.627966 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.628404 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.628507 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:30.632121 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:30.632127 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:30.632130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:30.632168 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:30.635205 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:30.636153 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:30.636158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:30.636161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:30.636333 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:30.636774 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:30.636776 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:30.636778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:30.637068 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:30.637196 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:30.637977 (+ 781us) server_negotiation.cc:299] Negotiation successful
0806 10:58:30.638176 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":257,"threads_started":1}
I20260806 10:58:30.639566 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:30.620381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:41633 (local address 127.30.218.3:52219)
0806 10:58:30.621238 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:30.621297 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:30.623400 (+ 2103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:30.624097 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:30.624106 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:30.624704 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:58:30.625328 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:30.625347 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.626471 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.626476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:30.627810 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:30.627825 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.628631 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.628637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:30.628859 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:30.629590 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:58:30.629617 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:58:30.631870 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:58:30.635366 (+ 3496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:58:30.635376 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:58:30.635393 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:58:30.636031 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:58:30.636443 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:58:30.636447 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:58:30.636450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:58:30.636670 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:58:30.637167 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:58:30.637173 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:58:30.637610 (+ 437us) client_negotiation.cc:769] Sending connection context
0806 10:58:30.637889 (+ 279us) client_negotiation.cc:240] Negotiation successful
0806 10:58:30.638343 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":331,"threads_started":1}
I20260806 10:58:30.641026 18290 heartbeater.cc:347] Connected to a master server at 127.30.218.62:41633
I20260806 10:58:30.641337 18290 heartbeater.cc:464] Registering TS with master...
I20260806 10:58:30.642138 18290 heartbeater.cc:511] Master 127.30.218.62:41633 requested a full tablet report, sending...
I20260806 10:58:30.643798 17816 ts_manager.cc:194] Registered new tserver with Master: b363584ad3be4198953a42d8ef1d24ac (127.30.218.3:37161)
I20260806 10:58:30.644658 17816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.3@KRBTEST.COM'} at 127.30.218.3:52219
I20260806 10:58:30.649108 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:58:30.666742 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:30.653744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48076 (local address 127.30.218.62:41633)
0806 10:58:30.653908 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:30.653914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:30.654365 (+ 451us) server_negotiation.cc:407] Connection header received
0806 10:58:30.654542 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:30.654547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:30.654627 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:30.654719 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:30.656214 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.656967 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.658366 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.658770 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.658881 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:30.661624 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:30.661631 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:30.661636 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:30.661674 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:30.663893 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:30.664719 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:30.664725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:30.664729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:30.664856 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:30.665278 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:30.665282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:30.665284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:30.665638 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:30.665897 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:30.666280 (+ 383us) server_negotiation.cc:299] Negotiation successful
0806 10:58:30.666466 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 10:58:30.670573 17816 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:58:30.677603 17816 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:58:30.713604 18303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:30.689814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:37161 (local address 127.0.0.1:44866)
0806 10:58:30.703726 (+ 13912us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:30.703805 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:30.704161 (+ 356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:30.705295 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:30.705303 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:30.705345 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:30.706486 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:30.706502 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.708409 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.708414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:30.710868 (+ 2454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:30.710883 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.712488 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.712494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:30.712698 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:30.712754 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 10:58:30.712886 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 10:58:30.713072 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13650,"mutex_wait_us":791,"spinlock_wait_cycles":4608,"thread_start_us":13404,"threads_started":1}
I20260806 10:58:30.713810 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:30.689891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44866 (local address 127.30.218.3:37161)
0806 10:58:30.703017 (+ 13126us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:30.703029 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:30.703960 (+ 931us) server_negotiation.cc:407] Connection header received
0806 10:58:30.704447 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:30.704457 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:30.704736 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:30.704917 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:30.706725 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.708278 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.711023 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.712295 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.712429 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:30.712672 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:30.712987 (+ 315us) server_negotiation.cc:299] Negotiation successful
0806 10:58:30.713159 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12937,"thread_start_us":12793,"threads_started":1}
I20260806 10:58:30.714663 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:30.704748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51798 (local address 127.30.218.1:43297)
0806 10:58:30.705901 (+ 1153us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:30.705909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:30.706647 (+ 738us) server_negotiation.cc:407] Connection header received
0806 10:58:30.706765 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:30.706773 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:30.707045 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:30.707240 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:30.707906 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.709491 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.711857 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.713030 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.713170 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:30.713392 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:30.714180 (+ 788us) server_negotiation.cc:299] Negotiation successful
0806 10:58:30.714335 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":978,"thread_start_us":794,"threads_started":1}
I20260806 10:58:30.714699 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:30.704336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:43297 (local address 127.0.0.1:51798)
0806 10:58:30.706334 (+ 1998us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:30.706355 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:30.706499 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:30.707388 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:30.707392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:30.707455 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:30.707754 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:30.707762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.709685 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.709689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:30.711717 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:30.711727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.713567 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.713573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:30.713736 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:30.713771 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 10:58:30.713851 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 10:58:30.713968 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1832,"mutex_wait_us":1,"thread_start_us":1595,"threads_started":1}
I20260806 10:58:30.714912 18304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:30.689653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50218 (local address 127.30.218.2:38495)
0806 10:58:30.702456 (+ 12803us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:30.702469 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:30.703959 (+ 1490us) server_negotiation.cc:407] Connection header received
0806 10:58:30.704447 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:30.704457 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:30.704749 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:30.704917 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:30.706236 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.708002 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.711545 (+ 3543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.712893 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.713029 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:30.713198 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:30.714466 (+ 1268us) server_negotiation.cc:299] Negotiation successful
0806 10:58:30.714630 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12546,"thread_start_us":13331,"threads_started":1}
I20260806 10:58:30.715667 18305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:30.689483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:38495 (local address 127.0.0.1:50218)
0806 10:58:30.703726 (+ 14243us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:30.703805 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:30.704161 (+ 356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:30.705295 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:30.705303 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:30.705345 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:30.705933 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:30.705951 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.708148 (+ 2197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.708153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:30.711352 (+ 3199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:30.711367 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.713347 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.713354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:30.713785 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:30.713826 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 10:58:30.715284 (+ 1458us) client_negotiation.cc:240] Negotiation successful
0806 10:58:30.715387 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13923,"mutex_wait_us":66,"spinlock_wait_cycles":20224,"thread_start_us":13859,"threads_started":1}
I20260806 10:58:30.718071 18223 tablet_service.cc:1467] Processing CreateTablet for tablet c40274747a48437ba5b867b00ed66ec1 (DEFAULT_TABLE table=test-table [id=a1acc5cc700d4a32a05d8852c3406551]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:58:30.718066 17945 tablet_service.cc:1467] Processing CreateTablet for tablet c40274747a48437ba5b867b00ed66ec1 (DEFAULT_TABLE table=test-table [id=a1acc5cc700d4a32a05d8852c3406551]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:58:30.718961 18083 tablet_service.cc:1467] Processing CreateTablet for tablet c40274747a48437ba5b867b00ed66ec1 (DEFAULT_TABLE table=test-table [id=a1acc5cc700d4a32a05d8852c3406551]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:58:30.719551 17945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c40274747a48437ba5b867b00ed66ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:30.719980 18223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c40274747a48437ba5b867b00ed66ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:30.720228 18083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c40274747a48437ba5b867b00ed66ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:30.731660 18309 tablet_bootstrap.cc:492] T c40274747a48437ba5b867b00ed66ec1 P 19f43c6d4d6b49f7840945eed04e37d8: Bootstrap starting.
I20260806 10:58:30.734813 18310 tablet_bootstrap.cc:492] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac: Bootstrap starting.
I20260806 10:58:30.735286 18309 tablet_bootstrap.cc:654] T c40274747a48437ba5b867b00ed66ec1 P 19f43c6d4d6b49f7840945eed04e37d8: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:30.736797 18309 log.cc:824] T c40274747a48437ba5b867b00ed66ec1 P 19f43c6d4d6b49f7840945eed04e37d8: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:30.737982 18311 tablet_bootstrap.cc:492] T c40274747a48437ba5b867b00ed66ec1 P 37a5cb5a1aa7472a878cff751ca9f3d9: Bootstrap starting.
I20260806 10:58:30.738730 18310 tablet_bootstrap.cc:654] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:30.739925 18310 log.cc:824] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:30.741842 18309 tablet_bootstrap.cc:492] T c40274747a48437ba5b867b00ed66ec1 P 19f43c6d4d6b49f7840945eed04e37d8: No bootstrap required, opened a new log
I20260806 10:58:30.741860 18311 tablet_bootstrap.cc:654] T c40274747a48437ba5b867b00ed66ec1 P 37a5cb5a1aa7472a878cff751ca9f3d9: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:30.742177 18309 ts_tablet_manager.cc:1397] T c40274747a48437ba5b867b00ed66ec1 P 19f43c6d4d6b49f7840945eed04e37d8: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260806 10:58:30.742944 18311 log.cc:824] T c40274747a48437ba5b867b00ed66ec1 P 37a5cb5a1aa7472a878cff751ca9f3d9: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:30.743407 18310 tablet_bootstrap.cc:492] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac: No bootstrap required, opened a new log
I20260806 10:58:30.743814 18310 ts_tablet_manager.cc:1397] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 10:58:30.745658 18311 tablet_bootstrap.cc:492] T c40274747a48437ba5b867b00ed66ec1 P 37a5cb5a1aa7472a878cff751ca9f3d9: No bootstrap required, opened a new log
I20260806 10:58:30.745927 18311 ts_tablet_manager.cc:1397] T c40274747a48437ba5b867b00ed66ec1 P 37a5cb5a1aa7472a878cff751ca9f3d9: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260806 10:58:30.753389 18309 raft_consensus.cc:348] T c40274747a48437ba5b867b00ed66ec1 P 19f43c6d4d6b49f7840945eed04e37d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a5cb5a1aa7472a878cff751ca9f3d9" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 38495 } } peers { permanent_uuid: "b363584ad3be4198953a42d8ef1d24ac" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 37161 } } peers { permanent_uuid: "19f43c6d4d6b49f7840945eed04e37d8" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43297 } }
I20260806 10:58:30.753973 18309 raft_consensus.cc:374] T c40274747a48437ba5b867b00ed66ec1 P 19f43c6d4d6b49f7840945eed04e37d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:30.753854 18311 raft_consensus.cc:348] T c40274747a48437ba5b867b00ed66ec1 P 37a5cb5a1aa7472a878cff751ca9f3d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a5cb5a1aa7472a878cff751ca9f3d9" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 38495 } } peers { permanent_uuid: "b363584ad3be4198953a42d8ef1d24ac" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 37161 } } peers { permanent_uuid: "19f43c6d4d6b49f7840945eed04e37d8" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43297 } }
I20260806 10:58:30.754139 18309 raft_consensus.cc:729] T c40274747a48437ba5b867b00ed66ec1 P 19f43c6d4d6b49f7840945eed04e37d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f43c6d4d6b49f7840945eed04e37d8, State: Initialized, Role: FOLLOWER
I20260806 10:58:30.754271 18311 raft_consensus.cc:374] T c40274747a48437ba5b867b00ed66ec1 P 37a5cb5a1aa7472a878cff751ca9f3d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:30.754375 18311 raft_consensus.cc:729] T c40274747a48437ba5b867b00ed66ec1 P 37a5cb5a1aa7472a878cff751ca9f3d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a5cb5a1aa7472a878cff751ca9f3d9, State: Initialized, Role: FOLLOWER
I20260806 10:58:30.754567 18310 raft_consensus.cc:348] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a5cb5a1aa7472a878cff751ca9f3d9" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 38495 } } peers { permanent_uuid: "b363584ad3be4198953a42d8ef1d24ac" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 37161 } } peers { permanent_uuid: "19f43c6d4d6b49f7840945eed04e37d8" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43297 } }
I20260806 10:58:30.755028 18310 raft_consensus.cc:374] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:30.754881 18309 consensus_queue.cc:260] T c40274747a48437ba5b867b00ed66ec1 P 19f43c6d4d6b49f7840945eed04e37d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a5cb5a1aa7472a878cff751ca9f3d9" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 38495 } } peers { permanent_uuid: "b363584ad3be4198953a42d8ef1d24ac" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 37161 } } peers { permanent_uuid: "19f43c6d4d6b49f7840945eed04e37d8" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43297 } }
I20260806 10:58:30.755134 18310 raft_consensus.cc:729] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b363584ad3be4198953a42d8ef1d24ac, State: Initialized, Role: FOLLOWER
I20260806 10:58:30.755009 18311 consensus_queue.cc:260] T c40274747a48437ba5b867b00ed66ec1 P 37a5cb5a1aa7472a878cff751ca9f3d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a5cb5a1aa7472a878cff751ca9f3d9" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 38495 } } peers { permanent_uuid: "b363584ad3be4198953a42d8ef1d24ac" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 37161 } } peers { permanent_uuid: "19f43c6d4d6b49f7840945eed04e37d8" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43297 } }
I20260806 10:58:30.755820 18310 consensus_queue.cc:260] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a5cb5a1aa7472a878cff751ca9f3d9" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 38495 } } peers { permanent_uuid: "b363584ad3be4198953a42d8ef1d24ac" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 37161 } } peers { permanent_uuid: "19f43c6d4d6b49f7840945eed04e37d8" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43297 } }
I20260806 10:58:30.757138 18290 heartbeater.cc:503] Master 127.30.218.62:41633 was elected leader, sending a full tablet report...
I20260806 10:58:30.757961 18310 ts_tablet_manager.cc:1428] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260806 10:58:30.759557 18309 ts_tablet_manager.cc:1428] T c40274747a48437ba5b867b00ed66ec1 P 19f43c6d4d6b49f7840945eed04e37d8: Time spent starting tablet: real 0.017s user 0.015s sys 0.002s
I20260806 10:58:30.761652 18151 heartbeater.cc:503] Master 127.30.218.62:41633 was elected leader, sending a full tablet report...
I20260806 10:58:30.762380 18311 ts_tablet_manager.cc:1428] T c40274747a48437ba5b867b00ed66ec1 P 37a5cb5a1aa7472a878cff751ca9f3d9: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
W20260806 10:58:30.773545 18013 tablet.cc:2369] T c40274747a48437ba5b867b00ed66ec1 P 19f43c6d4d6b49f7840945eed04e37d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:58:30.828933 18152 tablet.cc:2369] T c40274747a48437ba5b867b00ed66ec1 P 37a5cb5a1aa7472a878cff751ca9f3d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:58:30.868901 18291 tablet.cc:2369] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:58:30.930684 18317 raft_consensus.cc:482] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:58:30.930977 18317 raft_consensus.cc:504] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a5cb5a1aa7472a878cff751ca9f3d9" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 38495 } } peers { permanent_uuid: "b363584ad3be4198953a42d8ef1d24ac" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 37161 } } peers { permanent_uuid: "19f43c6d4d6b49f7840945eed04e37d8" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43297 } }
I20260806 10:58:30.932756 18317 leader_election.cc:288] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37a5cb5a1aa7472a878cff751ca9f3d9 (127.30.218.2:38495), 19f43c6d4d6b49f7840945eed04e37d8 (127.30.218.1:43297)
I20260806 10:58:30.940225 18304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:30.933131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:34577 (local address 127.30.218.2:38495)
0806 10:58:30.933346 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:30.933352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:30.933477 (+ 125us) server_negotiation.cc:407] Connection header received
0806 10:58:30.933761 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:30.933768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:30.933853 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:30.933961 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:30.934399 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.935408 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.937921 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.939212 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.939406 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:30.939492 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:30.939814 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 10:58:30.939952 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 10:58:30.940220 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:30.933009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:38495 (local address 127.30.218.3:34577)
0806 10:58:30.933345 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:30.933366 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:30.933543 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:30.933998 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:30.934003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:30.934037 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:30.934259 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:30.934271 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.935625 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.935631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:30.937727 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:30.937745 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.939387 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.939391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:30.939586 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:30.939625 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 10:58:30.939740 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 10:58:30.939899 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":2}
I20260806 10:58:30.940512 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:30.933308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:40759 (local address 127.30.218.1:43297)
0806 10:58:30.933549 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:30.933554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:30.934116 (+ 562us) server_negotiation.cc:407] Connection header received
0806 10:58:30.934283 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:30.934286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:30.934352 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:30.934447 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:30.935005 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.936035 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.938304 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.939530 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.939675 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:30.939750 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:30.940054 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 10:58:30.940233 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 10:58:30.940543 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:30.933260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:43297 (local address 127.30.218.3:40759)
0806 10:58:30.934025 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:30.934042 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:30.934159 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:30.934459 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:30.934462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:30.934491 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:30.934781 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:30.934791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.936243 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.936248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:30.938137 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:30.938147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:30.939734 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:30.939739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:30.939909 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:30.939940 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 10:58:30.940020 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 10:58:30.940151 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":21,"thread_start_us":458,"threads_started":1}
I20260806 10:58:30.941711 18103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c40274747a48437ba5b867b00ed66ec1" candidate_uuid: "b363584ad3be4198953a42d8ef1d24ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a5cb5a1aa7472a878cff751ca9f3d9" is_pre_election: true
I20260806 10:58:30.942339 18103 raft_consensus.cc:2454] T c40274747a48437ba5b867b00ed66ec1 P 37a5cb5a1aa7472a878cff751ca9f3d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b363584ad3be4198953a42d8ef1d24ac in term 0.
I20260806 10:58:30.941993 17965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c40274747a48437ba5b867b00ed66ec1" candidate_uuid: "b363584ad3be4198953a42d8ef1d24ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f43c6d4d6b49f7840945eed04e37d8" is_pre_election: true
I20260806 10:58:30.942793 17965 raft_consensus.cc:2454] T c40274747a48437ba5b867b00ed66ec1 P 19f43c6d4d6b49f7840945eed04e37d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b363584ad3be4198953a42d8ef1d24ac in term 0.
I20260806 10:58:30.943182 18176 leader_election.cc:302] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37a5cb5a1aa7472a878cff751ca9f3d9, b363584ad3be4198953a42d8ef1d24ac; no voters:
I20260806 10:58:30.943630 18317 raft_consensus.cc:2781] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:58:30.943779 18317 raft_consensus.cc:482] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:58:30.943883 18317 raft_consensus.cc:3037] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:30.947377 18317 raft_consensus.cc:504] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a5cb5a1aa7472a878cff751ca9f3d9" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 38495 } } peers { permanent_uuid: "b363584ad3be4198953a42d8ef1d24ac" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 37161 } } peers { permanent_uuid: "19f43c6d4d6b49f7840945eed04e37d8" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43297 } }
I20260806 10:58:30.948382 18317 leader_election.cc:288] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac [CANDIDATE]: Term 1 election: Requested vote from peers 37a5cb5a1aa7472a878cff751ca9f3d9 (127.30.218.2:38495), 19f43c6d4d6b49f7840945eed04e37d8 (127.30.218.1:43297)
I20260806 10:58:30.948768 17965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c40274747a48437ba5b867b00ed66ec1" candidate_uuid: "b363584ad3be4198953a42d8ef1d24ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f43c6d4d6b49f7840945eed04e37d8"
I20260806 10:58:30.949012 17965 raft_consensus.cc:3037] T c40274747a48437ba5b867b00ed66ec1 P 19f43c6d4d6b49f7840945eed04e37d8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:30.948999 18103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c40274747a48437ba5b867b00ed66ec1" candidate_uuid: "b363584ad3be4198953a42d8ef1d24ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a5cb5a1aa7472a878cff751ca9f3d9"
I20260806 10:58:30.949229 18103 raft_consensus.cc:3037] T c40274747a48437ba5b867b00ed66ec1 P 37a5cb5a1aa7472a878cff751ca9f3d9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:30.952219 17965 raft_consensus.cc:2454] T c40274747a48437ba5b867b00ed66ec1 P 19f43c6d4d6b49f7840945eed04e37d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b363584ad3be4198953a42d8ef1d24ac in term 1.
I20260806 10:58:30.952442 18103 raft_consensus.cc:2454] T c40274747a48437ba5b867b00ed66ec1 P 37a5cb5a1aa7472a878cff751ca9f3d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b363584ad3be4198953a42d8ef1d24ac in term 1.
I20260806 10:58:30.952944 18175 leader_election.cc:302] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f43c6d4d6b49f7840945eed04e37d8, b363584ad3be4198953a42d8ef1d24ac; no voters:
I20260806 10:58:30.953599 18317 raft_consensus.cc:2781] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:58:30.954131 18317 raft_consensus.cc:686] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac [term 1 LEADER]: Becoming Leader. State: Replica: b363584ad3be4198953a42d8ef1d24ac, State: Running, Role: LEADER
I20260806 10:58:30.954782 18317 consensus_queue.cc:237] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a5cb5a1aa7472a878cff751ca9f3d9" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 38495 } } peers { permanent_uuid: "b363584ad3be4198953a42d8ef1d24ac" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 37161 } } peers { permanent_uuid: "19f43c6d4d6b49f7840945eed04e37d8" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43297 } }
I20260806 10:58:30.961040 17815 catalog_manager.cc:5495] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac reported cstate change: term changed from 0 to 1, leader changed from <none> to b363584ad3be4198953a42d8ef1d24ac (127.30.218.3). New cstate: current_term: 1 leader_uuid: "b363584ad3be4198953a42d8ef1d24ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a5cb5a1aa7472a878cff751ca9f3d9" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 38495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b363584ad3be4198953a42d8ef1d24ac" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 37161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19f43c6d4d6b49f7840945eed04e37d8" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43297 } health_report { overall_health: UNKNOWN } } }
I20260806 10:58:31.018560 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:31.012996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44876 (local address 127.30.218.3:37161)
0806 10:58:31.013169 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:31.013174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:31.013262 (+ 88us) server_negotiation.cc:407] Connection header received
0806 10:58:31.013425 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:31.013429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:31.013582 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:31.013758 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:58:31.014119 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:31.014849 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:31.016352 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:31.016783 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:31.016880 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:31.017481 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:58:31.018006 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:58:31.018157 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:31.018292 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 10:58:31.018337 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 10:58:31.027343 18103 raft_consensus.cc:1261] T c40274747a48437ba5b867b00ed66ec1 P 37a5cb5a1aa7472a878cff751ca9f3d9 [term 1 FOLLOWER]: Refusing update from remote peer b363584ad3be4198953a42d8ef1d24ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:58:31.027341 17965 raft_consensus.cc:1261] T c40274747a48437ba5b867b00ed66ec1 P 19f43c6d4d6b49f7840945eed04e37d8 [term 1 FOLLOWER]: Refusing update from remote peer b363584ad3be4198953a42d8ef1d24ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:58:31.028352 18317 consensus_queue.cc:1035] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f43c6d4d6b49f7840945eed04e37d8" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 43297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:58:31.028717 18319 consensus_queue.cc:1035] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "37a5cb5a1aa7472a878cff751ca9f3d9" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 38495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:58:31.039361 18322 mvcc.cc:205] Tried to move back new op lower bound from 7315512979556503552 to 7315512979276034048. Current Snapshot: MvccSnapshot[applied={T|T < 7315512979556503552}]
I20260806 10:58:31.056851 17814 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:58:31.057197 17814 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:58:31.061394 17814 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a87b13e952e44b7abd96bf791c2e66b: Sending DeleteTablet for 3 replicas of tablet c40274747a48437ba5b867b00ed66ec1
I20260806 10:58:31.062742 18083 tablet_service.cc:1514] Processing DeleteTablet for tablet c40274747a48437ba5b867b00ed66ec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:58:31 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:50218
I20260806 10:58:31.062947 18223 tablet_service.cc:1514] Processing DeleteTablet for tablet c40274747a48437ba5b867b00ed66ec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:58:31 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:44866
I20260806 10:58:31.063575 17945 tablet_service.cc:1514] Processing DeleteTablet for tablet c40274747a48437ba5b867b00ed66ec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:58:31 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:51798
I20260806 10:58:31.064699 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 17882
I20260806 10:58:31.089738 18335 tablet_replica.cc:330] T c40274747a48437ba5b867b00ed66ec1 P 19f43c6d4d6b49f7840945eed04e37d8: stopping tablet replica
I20260806 10:58:31.090487 18335 raft_consensus.cc:2229] T c40274747a48437ba5b867b00ed66ec1 P 19f43c6d4d6b49f7840945eed04e37d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:58:31.091254 18333 tablet_replica.cc:330] T c40274747a48437ba5b867b00ed66ec1 P 37a5cb5a1aa7472a878cff751ca9f3d9: stopping tablet replica
I20260806 10:58:31.091887 18335 raft_consensus.cc:2258] T c40274747a48437ba5b867b00ed66ec1 P 19f43c6d4d6b49f7840945eed04e37d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:58:31.092438 18333 raft_consensus.cc:2229] T c40274747a48437ba5b867b00ed66ec1 P 37a5cb5a1aa7472a878cff751ca9f3d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:58:31.093357 18333 raft_consensus.cc:2258] T c40274747a48437ba5b867b00ed66ec1 P 37a5cb5a1aa7472a878cff751ca9f3d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:58:31.097488 18335 ts_tablet_manager.cc:1905] T c40274747a48437ba5b867b00ed66ec1 P 19f43c6d4d6b49f7840945eed04e37d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:58:31.098517 18333 ts_tablet_manager.cc:1905] T c40274747a48437ba5b867b00ed66ec1 P 37a5cb5a1aa7472a878cff751ca9f3d9: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 10:58:31 dist-test-slave-49h3 krb5kdc[17771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.1@KRBTEST.COM
I20260806 10:58:31.113190 18334 tablet_replica.cc:330] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac: stopping tablet replica
I20260806 10:58:31.113356 18335 ts_tablet_manager.cc:1918] T c40274747a48437ba5b867b00ed66ec1 P 19f43c6d4d6b49f7840945eed04e37d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:58:31.113597 18335 log.cc:1196] T c40274747a48437ba5b867b00ed66ec1 P 19f43c6d4d6b49f7840945eed04e37d8: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-0/wal/wals/c40274747a48437ba5b867b00ed66ec1
I20260806 10:58:31.113718 18334 raft_consensus.cc:2229] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:58:31.114185 18335 ts_tablet_manager.cc:1939] T c40274747a48437ba5b867b00ed66ec1 P 19f43c6d4d6b49f7840945eed04e37d8: Deleting consensus metadata
I20260806 10:58:31.114459 18334 raft_consensus.cc:2258] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:58:31.116761 17801 catalog_manager.cc:4841] TS 19f43c6d4d6b49f7840945eed04e37d8 (127.30.218.1:43297): tablet c40274747a48437ba5b867b00ed66ec1 (table test-table [id=a1acc5cc700d4a32a05d8852c3406551]) successfully deleted
I20260806 10:58:31.116950 18333 ts_tablet_manager.cc:1918] T c40274747a48437ba5b867b00ed66ec1 P 37a5cb5a1aa7472a878cff751ca9f3d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:58:31.117127 18333 log.cc:1196] T c40274747a48437ba5b867b00ed66ec1 P 37a5cb5a1aa7472a878cff751ca9f3d9: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/c40274747a48437ba5b867b00ed66ec1
I20260806 10:58:31.117751 18333 ts_tablet_manager.cc:1939] T c40274747a48437ba5b867b00ed66ec1 P 37a5cb5a1aa7472a878cff751ca9f3d9: Deleting consensus metadata
I20260806 10:58:31.119592 17802 catalog_manager.cc:4841] TS 37a5cb5a1aa7472a878cff751ca9f3d9 (127.30.218.2:38495): tablet c40274747a48437ba5b867b00ed66ec1 (table test-table [id=a1acc5cc700d4a32a05d8852c3406551]) successfully deleted
I20260806 10:58:31.121901 18334 ts_tablet_manager.cc:1905] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:58:31.123785 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:31.066369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51804 (local address 127.30.218.1:43297)
0806 10:58:31.066596 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:31.066603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:31.094653 (+ 28050us) server_negotiation.cc:407] Connection header received
0806 10:58:31.095274 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:31.095281 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:31.095380 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:31.095457 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:31.097440 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:31.098701 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:31.100810 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:31.101495 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:31.101617 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:31.115573 (+ 13956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:31.115590 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:31.115608 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:31.115666 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:31.120089 (+ 4423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:31.120960 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:31.120969 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:31.120973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:31.121135 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:31.121580 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:31.121585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:31.121589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:31.122012 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:31.122282 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:31.123355 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0806 10:58:31.123517 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 10:58:31.124395 18007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:31.129721 18334 ts_tablet_manager.cc:1918] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:58:31.129892 18334 log.cc:1196] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/c40274747a48437ba5b867b00ed66ec1
I20260806 10:58:31.130437 18334 ts_tablet_manager.cc:1939] T c40274747a48437ba5b867b00ed66ec1 P b363584ad3be4198953a42d8ef1d24ac: Deleting consensus metadata
I20260806 10:58:31.132153 17801 catalog_manager.cc:4841] TS b363584ad3be4198953a42d8ef1d24ac (127.30.218.3:37161): tablet c40274747a48437ba5b867b00ed66ec1 (table test-table [id=a1acc5cc700d4a32a05d8852c3406551]) successfully deleted
I20260806 10:58:31.221500 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 17882
I20260806 10:58:31.241191 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 18018
Aug 06 10:58:31 dist-test-slave-49h3 krb5kdc[17771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.2@KRBTEST.COM
I20260806 10:58:31.259162 18304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:31.242097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50224 (local address 127.30.218.2:38495)
0806 10:58:31.242299 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:31.242305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:31.242330 (+ 25us) server_negotiation.cc:407] Connection header received
0806 10:58:31.242403 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:31.242408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:31.242508 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:31.242644 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:31.243932 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:31.245083 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:31.246493 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:31.247147 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:31.247293 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:31.251833 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:31.251859 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:31.251882 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:31.251938 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:31.255850 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:31.256764 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:31.256774 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:31.256779 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:31.256962 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:31.257320 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:31.257325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:31.257329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:31.257907 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:31.258232 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:31.258505 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 10:58:31.258767 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 10:58:31.260030 18146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:31.349200 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 18018
I20260806 10:58:31.366698 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 18157
Aug 06 10:58:31 dist-test-slave-49h3 krb5kdc[17771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.3@KRBTEST.COM
I20260806 10:58:31.383934 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:31.367610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44892 (local address 127.30.218.3:37161)
0806 10:58:31.367802 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:31.367808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:31.367829 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:58:31.367950 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:31.367956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:31.368040 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:31.368135 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:31.369542 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:31.370442 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:31.371779 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:31.372278 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:31.372406 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:31.377166 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:31.377189 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:31.377205 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:31.377247 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:31.380506 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:31.381341 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:31.381350 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:31.381352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:31.381842 (+ 490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:31.382247 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:31.382253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:31.382256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:31.382693 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:31.382993 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:31.383317 (+ 324us) server_negotiation.cc:299] Negotiation successful
0806 10:58:31.383548 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 10:58:31.384677 18285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:31.510593 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 18157
I20260806 10:58:31.531004 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 17784
I20260806 10:58:31.532137 17850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:31.650264 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 17784
2026-08-06T10:58:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4673 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T10:58:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:58:31Z Disabled control of system clock
I20260806 10:58:31.744580 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:33791
--webserver_interface=127.30.218.62
--webserver_port=0
--builtin_ntp_servers=127.30.218.20:33391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:33791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 10:58:32.096344 18345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:32.096705 18345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:32.096817 18345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:32.104457 18345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:58:32.104614 18345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:32.104683 18345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:32.104766 18345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:58:32.104825 18345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:58:32.114540 18345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:33391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:33791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:33791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:32.115923 18345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:32.117901 18345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:32.129827 18353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:32.129869 18351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:32.129994 18350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:32.130889 18345 server_base.cc:1034] running on GCE node
I20260806 10:58:32.131676 18345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:32.133165 18345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:32.134366 18345 hybrid_clock.cc:648] HybridClock initialized: now 1786013912134317 us; error 45 us; skew 500 ppm
I20260806 10:58:32.134724 18345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:58:32.137015 18345 webserver.cc:460] Webserver started at http://127.30.218.62:43649/ using document root <none> and password file <none>
I20260806 10:58:32.137780 18345 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:32.137888 18345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:32.138217 18345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:32.141376 18345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "742e496f302941e98dd57a135d923230"
format_stamp: "Formatted at 2026-08-06 10:58:32 on dist-test-slave-49h3"
server_key: "4c1b72aa705804828451641f4be4d5b5"
server_key_iv: "e34da6bb3196b801c17fda50f14b6bc8"
server_key_version: "encryptionkey@0"
I20260806 10:58:32.142341 18345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "742e496f302941e98dd57a135d923230"
format_stamp: "Formatted at 2026-08-06 10:58:32 on dist-test-slave-49h3"
server_key: "4c1b72aa705804828451641f4be4d5b5"
server_key_iv: "e34da6bb3196b801c17fda50f14b6bc8"
server_key_version: "encryptionkey@0"
I20260806 10:58:32.148322 18345 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260806 10:58:32.152479 18359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:32.154137 18345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:58:32.154361 18345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "742e496f302941e98dd57a135d923230"
format_stamp: "Formatted at 2026-08-06 10:58:32 on dist-test-slave-49h3"
server_key: "4c1b72aa705804828451641f4be4d5b5"
server_key_iv: "e34da6bb3196b801c17fda50f14b6bc8"
server_key_version: "encryptionkey@0"
I20260806 10:58:32.154605 18345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:32.178225 18345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:32.179242 18345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:32.179551 18345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:32.199277 18345 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:33791
I20260806 10:58:32.199301 18410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:33791 every 8 connection(s)
I20260806 10:58:32.200804 18345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:58:32.204890 18411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:32.209075 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 18345
I20260806 10:58:32.209280 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:58:32.209903 31592 external_mini_cluster.cc:1428] Setting key 663158805a722ea8ae7b4e3561ceff9f
I20260806 10:58:32.218848 18411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 742e496f302941e98dd57a135d923230: Bootstrap starting.
I20260806 10:58:32.222404 18411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 742e496f302941e98dd57a135d923230: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:32.223785 18411 log.cc:824] T 00000000000000000000000000000000 P 742e496f302941e98dd57a135d923230: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:32.224897 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:32.212130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59806 (local address 127.30.218.62:33791)
0806 10:58:32.213262 (+ 1132us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:32.213274 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:32.213321 (+ 47us) server_negotiation.cc:407] Connection header received
0806 10:58:32.215117 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:32.215149 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:32.215739 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:32.216186 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:32.216868 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:32.218253 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:32.219730 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:32.220472 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:32.220672 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:32.221980 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:32.221989 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:32.222005 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:32.222632 (+ 627us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:32.222809 (+ 177us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:32.222839 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:32.222960 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:32.223207 (+ 247us) server_negotiation.cc:299] Negotiation successful
0806 10:58:32.223553 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":400,"threads_started":1}
I20260806 10:58:32.227288 18411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 742e496f302941e98dd57a135d923230: No bootstrap required, opened a new log
I20260806 10:58:32.234953 18411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 742e496f302941e98dd57a135d923230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742e496f302941e98dd57a135d923230" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 33791 } }
I20260806 10:58:32.235391 18411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 742e496f302941e98dd57a135d923230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:32.235486 18411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 742e496f302941e98dd57a135d923230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 742e496f302941e98dd57a135d923230, State: Initialized, Role: FOLLOWER
I20260806 10:58:32.236176 18411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 742e496f302941e98dd57a135d923230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742e496f302941e98dd57a135d923230" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 33791 } }
I20260806 10:58:32.236449 18411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 742e496f302941e98dd57a135d923230 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:58:32.236610 18411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 742e496f302941e98dd57a135d923230 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:58:32.236799 18411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 742e496f302941e98dd57a135d923230 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:32.239145 18411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 742e496f302941e98dd57a135d923230 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742e496f302941e98dd57a135d923230" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 33791 } }
I20260806 10:58:32.239691 18411 leader_election.cc:302] T 00000000000000000000000000000000 P 742e496f302941e98dd57a135d923230 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 742e496f302941e98dd57a135d923230; no voters:
I20260806 10:58:32.240307 18411 leader_election.cc:288] T 00000000000000000000000000000000 P 742e496f302941e98dd57a135d923230 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:58:32.240509 18417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 742e496f302941e98dd57a135d923230 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:58:32.241555 18417 raft_consensus.cc:686] T 00000000000000000000000000000000 P 742e496f302941e98dd57a135d923230 [term 1 LEADER]: Becoming Leader. State: Replica: 742e496f302941e98dd57a135d923230, State: Running, Role: LEADER
I20260806 10:58:32.242071 18417 consensus_queue.cc:237] T 00000000000000000000000000000000 P 742e496f302941e98dd57a135d923230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742e496f302941e98dd57a135d923230" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 33791 } }
I20260806 10:58:32.242545 18411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 742e496f302941e98dd57a135d923230 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:58:32.244944 18419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 742e496f302941e98dd57a135d923230 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 742e496f302941e98dd57a135d923230. Latest consensus state: current_term: 1 leader_uuid: "742e496f302941e98dd57a135d923230" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742e496f302941e98dd57a135d923230" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 33791 } } }
I20260806 10:58:32.244952 18418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 742e496f302941e98dd57a135d923230 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "742e496f302941e98dd57a135d923230" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742e496f302941e98dd57a135d923230" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 33791 } } }
I20260806 10:58:32.245471 18419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 742e496f302941e98dd57a135d923230 [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:32.245481 18418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 742e496f302941e98dd57a135d923230 [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:32.247483 18422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:58:32.252758 18422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:58:32.262830 18422 catalog_manager.cc:1347] Generated new cluster ID: b782d6743b184a399a6746444b9661e6
I20260806 10:58:32.262960 18422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:58:32.282891 18422 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:58:32.284289 18422 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:58:32.299471 18422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 742e496f302941e98dd57a135d923230: Generated new TSK 0
I20260806 10:58:32.300406 18422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:58:32.313297 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:33791
--builtin_ntp_servers=127.30.218.20:33391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 10:58:32.667586 18436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:32.667893 18436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:32.668007 18436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:32.675436 18436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:32.675547 18436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:32.675684 18436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:58:32.685356 18436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:33391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:33791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:32.686805 18436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:32.688645 18436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:32.701972 18441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:32.701947 18444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:32.702013 18442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:32.702697 18436 server_base.cc:1034] running on GCE node
I20260806 10:58:32.703317 18436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:32.704337 18436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:32.705600 18436 hybrid_clock.cc:648] HybridClock initialized: now 1786013912705540 us; error 53 us; skew 500 ppm
I20260806 10:58:32.705935 18436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:58:32.708303 18436 webserver.cc:460] Webserver started at http://127.30.218.1:37809/ using document root <none> and password file <none>
I20260806 10:58:32.709019 18436 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:32.709128 18436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:32.709470 18436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:32.712527 18436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "2af87fb5d2d54b188fa72bfc1146191a"
format_stamp: "Formatted at 2026-08-06 10:58:32 on dist-test-slave-49h3"
server_key: "ee552e5a2e8522e911b84a6ef55f513a"
server_key_iv: "964c0b51b4476d48b071b1fc69ece2b9"
server_key_version: "encryptionkey@0"
I20260806 10:58:32.713330 18436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "2af87fb5d2d54b188fa72bfc1146191a"
format_stamp: "Formatted at 2026-08-06 10:58:32 on dist-test-slave-49h3"
server_key: "ee552e5a2e8522e911b84a6ef55f513a"
server_key_iv: "964c0b51b4476d48b071b1fc69ece2b9"
server_key_version: "encryptionkey@0"
I20260806 10:58:32.718681 18436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 10:58:32.722366 18450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:32.723811 18436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:58:32.723994 18436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "2af87fb5d2d54b188fa72bfc1146191a"
format_stamp: "Formatted at 2026-08-06 10:58:32 on dist-test-slave-49h3"
server_key: "ee552e5a2e8522e911b84a6ef55f513a"
server_key_iv: "964c0b51b4476d48b071b1fc69ece2b9"
server_key_version: "encryptionkey@0"
I20260806 10:58:32.724227 18436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:32.738909 18436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:32.739732 18436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:32.740013 18436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:32.740927 18436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:58:32.742373 18436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:58:32.742475 18436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:32.742586 18436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:58:32.742655 18436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:32.772996 18436 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:35365
I20260806 10:58:32.773056 18569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:35365 every 8 connection(s)
I20260806 10:58:32.774345 18436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:58:32.782900 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 18436
I20260806 10:58:32.783056 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:58:32.783520 31592 external_mini_cluster.cc:1428] Setting key c47f047004af08c33b926044df757b10
I20260806 10:58:32.786839 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:33791
--builtin_ntp_servers=127.30.218.20:33391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 10:58:32.787434 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:32.776762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:36945 (local address 127.30.218.62:33791)
0806 10:58:32.777219 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:32.777224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:32.778341 (+ 1117us) server_negotiation.cc:407] Connection header received
0806 10:58:32.780204 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:32.780210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:32.780305 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:32.780405 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:32.781582 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:32.782353 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:32.784119 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:32.784682 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:32.784800 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:32.786543 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:32.786551 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:32.786555 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:32.786578 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:32.786634 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:32.786653 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:32.786751 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:32.786965 (+ 214us) server_negotiation.cc:299] Negotiation successful
0806 10:58:32.787079 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":231,"threads_started":1}
I20260806 10:58:32.788573 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:32.777232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:33791 (local address 127.30.218.1:36945)
0806 10:58:32.778167 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:32.778205 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:32.779908 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:32.780669 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:32.780679 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:32.780782 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:58:32.781387 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:32.781405 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:32.782523 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:32.782528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:32.783952 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:32.783966 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:32.784946 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:32.784955 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:32.785287 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:32.786186 (+ 899us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:58:32.786189 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:58:32.786238 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:58:32.786331 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:58:32.786747 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:58:32.786753 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:58:32.786759 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 10:58:32.786902 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 10:58:32.787333 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":297,"threads_started":1}
I20260806 10:58:32.790257 18570 heartbeater.cc:347] Connected to a master server at 127.30.218.62:33791
I20260806 10:58:32.790577 18570 heartbeater.cc:464] Registering TS with master...
I20260806 10:58:32.791505 18570 heartbeater.cc:511] Master 127.30.218.62:33791 requested a full tablet report, sending...
I20260806 10:58:32.794224 18376 ts_manager.cc:194] Registered new tserver with Master: 2af87fb5d2d54b188fa72bfc1146191a (127.30.218.1:35365)
I20260806 10:58:32.796293 18376 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.218.1:36945
W20260806 10:58:33.132762 18574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:33.133034 18574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:33.133159 18574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:33.140470 18574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:33.140599 18574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:33.140748 18574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:58:33.150950 18574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:33391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:33791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:33.152325 18574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:33.154270 18574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:33.166982 18579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:33.167004 18580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:33.166978 18582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:33.167842 18574 server_base.cc:1034] running on GCE node
I20260806 10:58:33.168444 18574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:33.169404 18574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:33.170650 18574 hybrid_clock.cc:648] HybridClock initialized: now 1786013913170607 us; error 35 us; skew 500 ppm
I20260806 10:58:33.170975 18574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:58:33.173347 18574 webserver.cc:460] Webserver started at http://127.30.218.2:34677/ using document root <none> and password file <none>
I20260806 10:58:33.174081 18574 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:33.174197 18574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:33.174516 18574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:33.177624 18574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "efe048dd5caa4a7caebc166a255fd3f6"
format_stamp: "Formatted at 2026-08-06 10:58:33 on dist-test-slave-49h3"
server_key: "351da8d799a641db5000d9a4f3f61793"
server_key_iv: "90ba780c1093975ca54626f511d733b4"
server_key_version: "encryptionkey@0"
I20260806 10:58:33.178505 18574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "efe048dd5caa4a7caebc166a255fd3f6"
format_stamp: "Formatted at 2026-08-06 10:58:33 on dist-test-slave-49h3"
server_key: "351da8d799a641db5000d9a4f3f61793"
server_key_iv: "90ba780c1093975ca54626f511d733b4"
server_key_version: "encryptionkey@0"
I20260806 10:58:33.184238 18574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:58:33.188139 18589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:33.189623 18574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:58:33.189806 18574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "efe048dd5caa4a7caebc166a255fd3f6"
format_stamp: "Formatted at 2026-08-06 10:58:33 on dist-test-slave-49h3"
server_key: "351da8d799a641db5000d9a4f3f61793"
server_key_iv: "90ba780c1093975ca54626f511d733b4"
server_key_version: "encryptionkey@0"
I20260806 10:58:33.190044 18574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:33.211302 18574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:33.212435 18574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:33.212788 18574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:33.214085 18574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:58:33.216086 18574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:58:33.216226 18574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:33.216344 18574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:58:33.216420 18574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:33.248935 18574 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:33163
I20260806 10:58:33.248981 18706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:33163 every 8 connection(s)
I20260806 10:58:33.250332 18574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:58:33.252215 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 18574
I20260806 10:58:33.252393 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:58:33.252851 31592 external_mini_cluster.cc:1428] Setting key 1f3782fdb38c6bf17a2af38ed9dc3db9
I20260806 10:58:33.256068 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:33791
--builtin_ntp_servers=127.30.218.20:33391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 10:58:33.265535 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:33.253009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:45727 (local address 127.30.218.62:33791)
0806 10:58:33.253232 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:33.253238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:33.255561 (+ 2323us) server_negotiation.cc:407] Connection header received
0806 10:58:33.257972 (+ 2411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:33.257976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:33.258047 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:33.258132 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:33.259551 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:33.260344 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:33.262571 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:33.263004 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:33.263118 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:33.264767 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:33.264772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:33.264776 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:33.264796 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:33.264830 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:33.264841 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:33.264913 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:33.265143 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 10:58:33.265243 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 10:58:33.266997 18709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:33.253608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:33791 (local address 127.30.218.2:45727)
0806 10:58:33.255400 (+ 1792us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:33.255450 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:33.257656 (+ 2206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:33.258438 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:33.258451 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:33.258583 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:58:33.259390 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:33.259413 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:33.260516 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:33.260521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:33.262374 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:33.262391 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:33.263259 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:33.263266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:33.263573 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:33.264364 (+ 791us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:58:33.264367 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:58:33.264428 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:58:33.264559 (+ 131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:58:33.264929 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:58:33.264939 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:58:33.264947 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 10:58:33.265131 (+ 184us) client_negotiation.cc:240] Negotiation successful
0806 10:58:33.265618 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1201,"thread_start_us":983,"threads_started":1}
I20260806 10:58:33.268611 18707 heartbeater.cc:347] Connected to a master server at 127.30.218.62:33791
I20260806 10:58:33.268920 18707 heartbeater.cc:464] Registering TS with master...
I20260806 10:58:33.269760 18707 heartbeater.cc:511] Master 127.30.218.62:33791 requested a full tablet report, sending...
I20260806 10:58:33.271185 18376 ts_manager.cc:194] Registered new tserver with Master: efe048dd5caa4a7caebc166a255fd3f6 (127.30.218.2:33163)
I20260806 10:58:33.272011 18376 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.218.2:45727
W20260806 10:58:33.597869 18710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:33.598186 18710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:33.598304 18710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:33.605870 18710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:33.605995 18710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:33.606164 18710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:58:33.615818 18710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:33391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:33791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:33.617265 18710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:33.619242 18710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:33.632483 18718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:33.632550 18716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:33.632582 18715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:33.633430 18710 server_base.cc:1034] running on GCE node
I20260806 10:58:33.634122 18710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:33.635157 18710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:33.636369 18710 hybrid_clock.cc:648] HybridClock initialized: now 1786013913636317 us; error 48 us; skew 500 ppm
I20260806 10:58:33.636694 18710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:58:33.639134 18710 webserver.cc:460] Webserver started at http://127.30.218.3:45561/ using document root <none> and password file <none>
I20260806 10:58:33.639900 18710 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:33.640012 18710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:33.640321 18710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:33.643406 18710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "916c04b37bdf475bac1fca90f1deb077"
format_stamp: "Formatted at 2026-08-06 10:58:33 on dist-test-slave-49h3"
server_key: "4a2eaef498851912d737507a1001c1af"
server_key_iv: "ba738539255f127123633726dd997dc6"
server_key_version: "encryptionkey@0"
I20260806 10:58:33.644256 18710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "916c04b37bdf475bac1fca90f1deb077"
format_stamp: "Formatted at 2026-08-06 10:58:33 on dist-test-slave-49h3"
server_key: "4a2eaef498851912d737507a1001c1af"
server_key_iv: "ba738539255f127123633726dd997dc6"
server_key_version: "encryptionkey@0"
I20260806 10:58:33.649869 18710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260806 10:58:33.653666 18724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:33.655112 18710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:58:33.655292 18710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "916c04b37bdf475bac1fca90f1deb077"
format_stamp: "Formatted at 2026-08-06 10:58:33 on dist-test-slave-49h3"
server_key: "4a2eaef498851912d737507a1001c1af"
server_key_iv: "ba738539255f127123633726dd997dc6"
server_key_version: "encryptionkey@0"
I20260806 10:58:33.655521 18710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:33.672447 18710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:33.673352 18710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:33.673728 18710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:33.674831 18710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:58:33.676397 18710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:58:33.676502 18710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:33.676620 18710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:58:33.676689 18710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:33.707837 18710 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:39443
I20260806 10:58:33.707878 18840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:39443 every 8 connection(s)
I20260806 10:58:33.709122 18710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:58:33.710956 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 18710
I20260806 10:58:33.711145 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:58:33.711699 31592 external_mini_cluster.cc:1428] Setting key 600484deb2af3338fd1d7a503a2beb85
I20260806 10:58:33.721735 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:33.711624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:58213 (local address 127.30.218.62:33791)
0806 10:58:33.711799 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:33.711804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:33.713406 (+ 1602us) server_negotiation.cc:407] Connection header received
0806 10:58:33.715501 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:33.715506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:33.715590 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:33.715684 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:33.716732 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:33.717505 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:33.719037 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:33.719444 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:33.719558 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:33.720859 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:33.720864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:33.720867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:33.720889 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:33.720922 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:33.720933 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:33.721008 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:33.721223 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 10:58:33.721329 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 10:58:33.722776 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:33.712273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:33791 (local address 127.30.218.3:58213)
0806 10:58:33.713273 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:33.713317 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:33.715248 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:33.715919 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:33.715927 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:33.716012 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:58:33.716571 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:33.716587 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:33.717646 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:33.717650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:33.718897 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:33.718910 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:33.719665 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:33.719673 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:33.719897 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:33.720536 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:58:33.720538 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:58:33.720578 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:58:33.720661 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:58:33.721019 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:58:33.721032 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:58:33.721039 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 10:58:33.721210 (+ 171us) client_negotiation.cc:240] Negotiation successful
0806 10:58:33.721646 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":291,"threads_started":1}
I20260806 10:58:33.724118 18841 heartbeater.cc:347] Connected to a master server at 127.30.218.62:33791
I20260806 10:58:33.724399 18841 heartbeater.cc:464] Registering TS with master...
I20260806 10:58:33.725150 18841 heartbeater.cc:511] Master 127.30.218.62:33791 requested a full tablet report, sending...
I20260806 10:58:33.726873 18376 ts_manager.cc:194] Registered new tserver with Master: 916c04b37bdf475bac1fca90f1deb077 (127.30.218.3:39443)
I20260806 10:58:33.727964 18376 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.218.3:58213
I20260806 10:58:33.739837 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:58:33.750224 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:33.744225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59814 (local address 127.30.218.62:33791)
0806 10:58:33.744397 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:33.744401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:33.744771 (+ 370us) server_negotiation.cc:407] Connection header received
0806 10:58:33.744918 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:33.744922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:33.744990 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:33.745082 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:33.745631 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:33.746417 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:33.747900 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:33.748357 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:33.748462 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:33.749592 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:33.749598 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:33.749601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:33.749622 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:33.749657 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:33.749669 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:33.749790 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:33.749920 (+ 130us) server_negotiation.cc:299] Negotiation successful
0806 10:58:33.749970 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 10:58:33.754240 18376 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:58:33.761816 18376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:58:33.791541 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:33.775798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41496 (local address 127.30.218.3:39443)
0806 10:58:33.782443 (+ 6645us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:33.782452 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:33.782902 (+ 450us) server_negotiation.cc:407] Connection header received
0806 10:58:33.783023 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:33.783031 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:33.783304 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:33.783515 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:33.784523 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:33.786157 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:33.789254 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:33.790262 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:33.790391 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:33.790701 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:33.790992 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 10:58:33.791182 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6482,"thread_start_us":6330,"threads_started":1}
I20260806 10:58:33.791733 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:33.775210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:39443 (local address 127.0.0.1:41496)
0806 10:58:33.782585 (+ 7375us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:33.782605 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:33.782755 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:33.783777 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:33.783783 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:33.783808 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:33.784280 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:33.784292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:33.786309 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:33.786317 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:33.789073 (+ 2756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:33.789084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:33.790461 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:33.790469 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:33.790709 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:33.790775 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 10:58:33.790916 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 10:58:33.791169 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7237,"mutex_wait_us":546,"thread_start_us":6257,"threads_started":1}
I20260806 10:58:33.792152 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:33.776814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55378 (local address 127.30.218.1:35365)
0806 10:58:33.781891 (+ 5077us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:33.781901 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:33.783679 (+ 1778us) server_negotiation.cc:407] Connection header received
0806 10:58:33.783797 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:33.783804 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:33.784016 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:33.784219 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:33.784756 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:33.785834 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:33.788862 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:33.790395 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:33.790532 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:33.790803 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:33.791356 (+ 553us) server_negotiation.cc:299] Negotiation successful
0806 10:58:33.791796 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4882,"thread_start_us":4749,"threads_started":1}
I20260806 10:58:33.792824 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:33.774865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:33163 (local address 127.0.0.1:51442)
0806 10:58:33.780997 (+ 6132us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:33.781058 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:33.781270 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:33.783540 (+ 2270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:33.783547 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:33.783575 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:33.784136 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:33.784156 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:33.786250 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:33.786255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:33.788809 (+ 2554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:33.788825 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:33.790628 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:33.790636 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:33.790821 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:33.790854 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 10:58:33.790933 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 10:58:33.791175 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5817,"mutex_wait_us":32,"thread_start_us":8254,"threads_started":1}
I20260806 10:58:33.792894 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:33.776502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:35365 (local address 127.0.0.1:55378)
0806 10:58:33.783376 (+ 6874us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:33.783393 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:33.783523 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:33.784231 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:33.784235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:33.784260 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:33.784595 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:33.784604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:33.785971 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:33.785977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:33.788631 (+ 2654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:33.788654 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:33.790932 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:33.790939 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:33.791141 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:33.791170 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 10:58:33.792434 (+ 1264us) client_negotiation.cc:240] Negotiation successful
0806 10:58:33.792565 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6757,"mutex_wait_us":725,"thread_start_us":4011,"threads_started":1}
I20260806 10:58:33.793985 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:33.775497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51442 (local address 127.30.218.2:33163)
0806 10:58:33.782881 (+ 7384us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:33.782889 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:33.782913 (+ 24us) server_negotiation.cc:407] Connection header received
0806 10:58:33.783023 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:33.783031 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:33.783304 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:33.783519 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:33.784435 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:33.786077 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:33.789000 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:33.790395 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:33.793191 (+ 2796us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:33.793407 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:33.793516 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 10:58:33.793665 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7222,"thread_start_us":7116,"threads_started":1}
I20260806 10:58:33.796262 18502 tablet_service.cc:1467] Processing CreateTablet for tablet 12ec51dc39e84036a0e5993a9a6cf005 (DEFAULT_TABLE table=test-table [id=ecd78263bf7848d19870ea6971804fa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:58:33.796160 18773 tablet_service.cc:1467] Processing CreateTablet for tablet 12ec51dc39e84036a0e5993a9a6cf005 (DEFAULT_TABLE table=test-table [id=ecd78263bf7848d19870ea6971804fa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:58:33.797533 18502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12ec51dc39e84036a0e5993a9a6cf005. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:33.797650 18773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12ec51dc39e84036a0e5993a9a6cf005. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:33.797300 18638 tablet_service.cc:1467] Processing CreateTablet for tablet 12ec51dc39e84036a0e5993a9a6cf005 (DEFAULT_TABLE table=test-table [id=ecd78263bf7848d19870ea6971804fa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:58:33.798540 18638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12ec51dc39e84036a0e5993a9a6cf005. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:33.800311 18570 heartbeater.cc:503] Master 127.30.218.62:33791 was elected leader, sending a full tablet report...
I20260806 10:58:33.810470 18859 tablet_bootstrap.cc:492] T 12ec51dc39e84036a0e5993a9a6cf005 P 2af87fb5d2d54b188fa72bfc1146191a: Bootstrap starting.
I20260806 10:58:33.812106 18860 tablet_bootstrap.cc:492] T 12ec51dc39e84036a0e5993a9a6cf005 P 916c04b37bdf475bac1fca90f1deb077: Bootstrap starting.
I20260806 10:58:33.815563 18859 tablet_bootstrap.cc:654] T 12ec51dc39e84036a0e5993a9a6cf005 P 2af87fb5d2d54b188fa72bfc1146191a: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:33.815864 18860 tablet_bootstrap.cc:654] T 12ec51dc39e84036a0e5993a9a6cf005 P 916c04b37bdf475bac1fca90f1deb077: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:33.817315 18860 log.cc:824] T 12ec51dc39e84036a0e5993a9a6cf005 P 916c04b37bdf475bac1fca90f1deb077: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:33.817315 18859 log.cc:824] T 12ec51dc39e84036a0e5993a9a6cf005 P 2af87fb5d2d54b188fa72bfc1146191a: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:33.820721 18859 tablet_bootstrap.cc:492] T 12ec51dc39e84036a0e5993a9a6cf005 P 2af87fb5d2d54b188fa72bfc1146191a: No bootstrap required, opened a new log
I20260806 10:58:33.821023 18861 tablet_bootstrap.cc:492] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6: Bootstrap starting.
I20260806 10:58:33.821062 18859 ts_tablet_manager.cc:1397] T 12ec51dc39e84036a0e5993a9a6cf005 P 2af87fb5d2d54b188fa72bfc1146191a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260806 10:58:33.821501 18860 tablet_bootstrap.cc:492] T 12ec51dc39e84036a0e5993a9a6cf005 P 916c04b37bdf475bac1fca90f1deb077: No bootstrap required, opened a new log
I20260806 10:58:33.821785 18860 ts_tablet_manager.cc:1397] T 12ec51dc39e84036a0e5993a9a6cf005 P 916c04b37bdf475bac1fca90f1deb077: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260806 10:58:33.824040 18861 tablet_bootstrap.cc:654] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:33.825074 18861 log.cc:824] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:33.828226 18861 tablet_bootstrap.cc:492] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6: No bootstrap required, opened a new log
I20260806 10:58:33.828505 18861 ts_tablet_manager.cc:1397] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.003s
I20260806 10:58:33.832263 18860 raft_consensus.cc:348] T 12ec51dc39e84036a0e5993a9a6cf005 P 916c04b37bdf475bac1fca90f1deb077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe048dd5caa4a7caebc166a255fd3f6" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33163 } } peers { permanent_uuid: "916c04b37bdf475bac1fca90f1deb077" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 39443 } } peers { permanent_uuid: "2af87fb5d2d54b188fa72bfc1146191a" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 35365 } }
I20260806 10:58:33.832778 18860 raft_consensus.cc:374] T 12ec51dc39e84036a0e5993a9a6cf005 P 916c04b37bdf475bac1fca90f1deb077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:33.832890 18860 raft_consensus.cc:729] T 12ec51dc39e84036a0e5993a9a6cf005 P 916c04b37bdf475bac1fca90f1deb077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916c04b37bdf475bac1fca90f1deb077, State: Initialized, Role: FOLLOWER
I20260806 10:58:33.833285 18859 raft_consensus.cc:348] T 12ec51dc39e84036a0e5993a9a6cf005 P 2af87fb5d2d54b188fa72bfc1146191a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe048dd5caa4a7caebc166a255fd3f6" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33163 } } peers { permanent_uuid: "916c04b37bdf475bac1fca90f1deb077" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 39443 } } peers { permanent_uuid: "2af87fb5d2d54b188fa72bfc1146191a" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 35365 } }
I20260806 10:58:33.833755 18859 raft_consensus.cc:374] T 12ec51dc39e84036a0e5993a9a6cf005 P 2af87fb5d2d54b188fa72bfc1146191a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:33.833619 18860 consensus_queue.cc:260] T 12ec51dc39e84036a0e5993a9a6cf005 P 916c04b37bdf475bac1fca90f1deb077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe048dd5caa4a7caebc166a255fd3f6" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33163 } } peers { permanent_uuid: "916c04b37bdf475bac1fca90f1deb077" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 39443 } } peers { permanent_uuid: "2af87fb5d2d54b188fa72bfc1146191a" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 35365 } }
I20260806 10:58:33.833858 18859 raft_consensus.cc:729] T 12ec51dc39e84036a0e5993a9a6cf005 P 2af87fb5d2d54b188fa72bfc1146191a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2af87fb5d2d54b188fa72bfc1146191a, State: Initialized, Role: FOLLOWER
I20260806 10:58:33.834507 18859 consensus_queue.cc:260] T 12ec51dc39e84036a0e5993a9a6cf005 P 2af87fb5d2d54b188fa72bfc1146191a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe048dd5caa4a7caebc166a255fd3f6" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33163 } } peers { permanent_uuid: "916c04b37bdf475bac1fca90f1deb077" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 39443 } } peers { permanent_uuid: "2af87fb5d2d54b188fa72bfc1146191a" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 35365 } }
I20260806 10:58:33.836515 18859 ts_tablet_manager.cc:1428] T 12ec51dc39e84036a0e5993a9a6cf005 P 2af87fb5d2d54b188fa72bfc1146191a: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260806 10:58:33.836396 18861 raft_consensus.cc:348] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe048dd5caa4a7caebc166a255fd3f6" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33163 } } peers { permanent_uuid: "916c04b37bdf475bac1fca90f1deb077" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 39443 } } peers { permanent_uuid: "2af87fb5d2d54b188fa72bfc1146191a" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 35365 } }
I20260806 10:58:33.836818 18861 raft_consensus.cc:374] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:33.836915 18861 raft_consensus.cc:729] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efe048dd5caa4a7caebc166a255fd3f6, State: Initialized, Role: FOLLOWER
I20260806 10:58:33.837517 18861 consensus_queue.cc:260] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe048dd5caa4a7caebc166a255fd3f6" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33163 } } peers { permanent_uuid: "916c04b37bdf475bac1fca90f1deb077" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 39443 } } peers { permanent_uuid: "2af87fb5d2d54b188fa72bfc1146191a" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 35365 } }
I20260806 10:58:33.837702 18841 heartbeater.cc:503] Master 127.30.218.62:33791 was elected leader, sending a full tablet report...
I20260806 10:58:33.838214 18860 ts_tablet_manager.cc:1428] T 12ec51dc39e84036a0e5993a9a6cf005 P 916c04b37bdf475bac1fca90f1deb077: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260806 10:58:33.839673 18707 heartbeater.cc:503] Master 127.30.218.62:33791 was elected leader, sending a full tablet report...
I20260806 10:58:33.840323 18861 ts_tablet_manager.cc:1428] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
W20260806 10:58:33.960778 18842 tablet.cc:2369] T 12ec51dc39e84036a0e5993a9a6cf005 P 916c04b37bdf475bac1fca90f1deb077: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:58:34.002408 18708 tablet.cc:2369] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:58:34.005506 18867 raft_consensus.cc:482] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:58:34.005846 18867 raft_consensus.cc:504] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe048dd5caa4a7caebc166a255fd3f6" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33163 } } peers { permanent_uuid: "916c04b37bdf475bac1fca90f1deb077" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 39443 } } peers { permanent_uuid: "2af87fb5d2d54b188fa72bfc1146191a" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 35365 } }
I20260806 10:58:34.007612 18867 leader_election.cc:288] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 916c04b37bdf475bac1fca90f1deb077 (127.30.218.3:39443), 2af87fb5d2d54b188fa72bfc1146191a (127.30.218.1:35365)
I20260806 10:58:34.009783 18866 raft_consensus.cc:482] T 12ec51dc39e84036a0e5993a9a6cf005 P 2af87fb5d2d54b188fa72bfc1146191a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:58:34.010116 18866 raft_consensus.cc:504] T 12ec51dc39e84036a0e5993a9a6cf005 P 2af87fb5d2d54b188fa72bfc1146191a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe048dd5caa4a7caebc166a255fd3f6" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33163 } } peers { permanent_uuid: "916c04b37bdf475bac1fca90f1deb077" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 39443 } } peers { permanent_uuid: "2af87fb5d2d54b188fa72bfc1146191a" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 35365 } }
I20260806 10:58:34.011992 18866 leader_election.cc:288] T 12ec51dc39e84036a0e5993a9a6cf005 P 2af87fb5d2d54b188fa72bfc1146191a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efe048dd5caa4a7caebc166a255fd3f6 (127.30.218.2:33163), 916c04b37bdf475bac1fca90f1deb077 (127.30.218.3:39443)
I20260806 10:58:34.016535 18869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:34.008261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:39443 (local address 127.30.218.2:55287)
0806 10:58:34.009150 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:34.009170 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:34.009299 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:34.009756 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:34.009760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:34.009799 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:34.010255 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:34.010268 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:34.011575 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:34.011581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:34.014213 (+ 2632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:34.014232 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:34.015738 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:34.015745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:34.015898 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:34.015931 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 10:58:34.016022 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 10:58:34.016166 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"mutex_wait_us":224,"thread_start_us":487,"threads_started":1}
I20260806 10:58:34.016920 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:34.008257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:54511 (local address 127.30.218.1:35365)
0806 10:58:34.008494 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:34.008500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:34.009179 (+ 679us) server_negotiation.cc:407] Connection header received
0806 10:58:34.009368 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:34.009372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:34.009481 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:34.009581 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:34.010227 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:34.011355 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:34.014266 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:34.015660 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:34.015801 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:34.015873 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:34.016484 (+ 611us) server_negotiation.cc:299] Negotiation successful
0806 10:58:34.016650 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260806 10:58:34.017211 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:34.008150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:55287 (local address 127.30.218.3:39443)
0806 10:58:34.008368 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:34.008374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:34.009265 (+ 891us) server_negotiation.cc:407] Connection header received
0806 10:58:34.009437 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:34.009465 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:34.009540 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:34.009624 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:34.010433 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:34.011458 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:34.014396 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:34.015543 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:34.015694 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:34.015767 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:34.016843 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0806 10:58:34.016976 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 10:58:34.018178 18868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:34.008196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:35365 (local address 127.30.218.2:54511)
0806 10:58:34.009052 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:34.009076 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:34.009232 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:34.009598 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:34.009603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:34.009627 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:34.010075 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:34.010092 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:34.011504 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:34.011510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:34.014010 (+ 2500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:34.014028 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:34.016059 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:34.016065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:34.016260 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:34.016295 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 10:58:34.017598 (+ 1303us) client_negotiation.cc:240] Negotiation successful
0806 10:58:34.017766 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":6,"spinlock_wait_cycles":19328,"thread_start_us":521,"threads_started":1}
I20260806 10:58:34.018324 18794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ec51dc39e84036a0e5993a9a6cf005" candidate_uuid: "efe048dd5caa4a7caebc166a255fd3f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916c04b37bdf475bac1fca90f1deb077" is_pre_election: true
I20260806 10:58:34.018978 18794 raft_consensus.cc:2454] T 12ec51dc39e84036a0e5993a9a6cf005 P 916c04b37bdf475bac1fca90f1deb077 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efe048dd5caa4a7caebc166a255fd3f6 in term 0.
I20260806 10:58:34.019668 18523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ec51dc39e84036a0e5993a9a6cf005" candidate_uuid: "efe048dd5caa4a7caebc166a255fd3f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2af87fb5d2d54b188fa72bfc1146191a" is_pre_election: true
I20260806 10:58:34.019814 18591 leader_election.cc:302] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 916c04b37bdf475bac1fca90f1deb077, efe048dd5caa4a7caebc166a255fd3f6; no voters:
I20260806 10:58:34.020155 18523 raft_consensus.cc:2454] T 12ec51dc39e84036a0e5993a9a6cf005 P 2af87fb5d2d54b188fa72bfc1146191a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efe048dd5caa4a7caebc166a255fd3f6 in term 0.
I20260806 10:58:34.020833 18867 raft_consensus.cc:2781] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:58:34.021056 18867 raft_consensus.cc:482] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:58:34.021162 18867 raft_consensus.cc:3037] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:34.024240 18867 raft_consensus.cc:504] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe048dd5caa4a7caebc166a255fd3f6" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33163 } } peers { permanent_uuid: "916c04b37bdf475bac1fca90f1deb077" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 39443 } } peers { permanent_uuid: "2af87fb5d2d54b188fa72bfc1146191a" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 35365 } }
I20260806 10:58:34.025372 18867 leader_election.cc:288] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6 [CANDIDATE]: Term 1 election: Requested vote from peers 916c04b37bdf475bac1fca90f1deb077 (127.30.218.3:39443), 2af87fb5d2d54b188fa72bfc1146191a (127.30.218.1:35365)
I20260806 10:58:34.025480 18794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ec51dc39e84036a0e5993a9a6cf005" candidate_uuid: "efe048dd5caa4a7caebc166a255fd3f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916c04b37bdf475bac1fca90f1deb077"
I20260806 10:58:34.025528 18523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ec51dc39e84036a0e5993a9a6cf005" candidate_uuid: "efe048dd5caa4a7caebc166a255fd3f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2af87fb5d2d54b188fa72bfc1146191a"
I20260806 10:58:34.025717 18794 raft_consensus.cc:3037] T 12ec51dc39e84036a0e5993a9a6cf005 P 916c04b37bdf475bac1fca90f1deb077 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:34.025758 18523 raft_consensus.cc:3037] T 12ec51dc39e84036a0e5993a9a6cf005 P 2af87fb5d2d54b188fa72bfc1146191a [term 0 FOLLOWER]: Advancing to term 1
W20260806 10:58:34.028247 18571 tablet.cc:2369] T 12ec51dc39e84036a0e5993a9a6cf005 P 2af87fb5d2d54b188fa72bfc1146191a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:58:34.029227 18794 raft_consensus.cc:2454] T 12ec51dc39e84036a0e5993a9a6cf005 P 916c04b37bdf475bac1fca90f1deb077 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efe048dd5caa4a7caebc166a255fd3f6 in term 1.
I20260806 10:58:34.029763 18591 leader_election.cc:302] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 916c04b37bdf475bac1fca90f1deb077, efe048dd5caa4a7caebc166a255fd3f6; no voters:
I20260806 10:58:34.030138 18870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:34.012607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:39443 (local address 127.30.218.1:41973)
0806 10:58:34.022443 (+ 9836us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:34.022462 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:34.022595 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:34.022917 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:34.022922 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:34.022950 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:34.023416 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:34.023430 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:34.024827 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:34.024832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:34.027227 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:34.027246 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:34.029264 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:34.029271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:34.029472 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:34.029506 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 10:58:34.029590 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 10:58:34.029745 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9720,"mutex_wait_us":3,"thread_start_us":9013,"threads_started":1}
I20260806 10:58:34.030216 18867 raft_consensus.cc:2781] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:58:34.030685 18523 raft_consensus.cc:2454] T 12ec51dc39e84036a0e5993a9a6cf005 P 2af87fb5d2d54b188fa72bfc1146191a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efe048dd5caa4a7caebc166a255fd3f6 in term 1.
I20260806 10:58:34.030925 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:34.013594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:41973 (local address 127.30.218.3:39443)
0806 10:58:34.014867 (+ 1273us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:34.014873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:34.022558 (+ 7685us) server_negotiation.cc:407] Connection header received
0806 10:58:34.022709 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:34.022714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:34.022781 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:34.022881 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:34.023616 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:34.024700 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:34.027495 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:34.028855 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:34.029002 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:34.029071 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:34.030366 (+ 1295us) server_negotiation.cc:299] Negotiation successful
0806 10:58:34.030518 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1130,"thread_start_us":898,"threads_started":1}
I20260806 10:58:34.030934 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:34.013139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:33279 (local address 127.30.218.2:33163)
0806 10:58:34.013430 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:34.013435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:34.022156 (+ 8721us) server_negotiation.cc:407] Connection header received
0806 10:58:34.022407 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:34.022412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:34.022501 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:34.022583 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:34.023351 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:34.024440 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:34.028093 (+ 3653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:34.029282 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:34.029431 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:34.029536 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:34.030499 (+ 963us) server_negotiation.cc:299] Negotiation successful
0806 10:58:34.030661 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260806 10:58:34.031441 18794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ec51dc39e84036a0e5993a9a6cf005" candidate_uuid: "2af87fb5d2d54b188fa72bfc1146191a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916c04b37bdf475bac1fca90f1deb077" is_pre_election: true
I20260806 10:58:34.031771 18794 raft_consensus.cc:2379] T 12ec51dc39e84036a0e5993a9a6cf005 P 916c04b37bdf475bac1fca90f1deb077 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2af87fb5d2d54b188fa72bfc1146191a in current term 1: Already voted for candidate efe048dd5caa4a7caebc166a255fd3f6 in this term.
I20260806 10:58:34.031735 18867 raft_consensus.cc:686] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6 [term 1 LEADER]: Becoming Leader. State: Replica: efe048dd5caa4a7caebc166a255fd3f6, State: Running, Role: LEADER
I20260806 10:58:34.032051 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:34.012156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:33163 (local address 127.30.218.1:33279)
0806 10:58:34.022036 (+ 9880us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:34.022060 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:34.022275 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:34.022729 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:34.022734 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:34.022758 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:34.023207 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:34.023219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:34.024598 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:34.024603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:34.027800 (+ 3197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:34.027819 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:34.029862 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:34.029868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:34.030108 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:34.030143 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 10:58:34.031544 (+ 1401us) client_negotiation.cc:240] Negotiation successful
0806 10:58:34.031711 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9709,"mutex_wait_us":380,"thread_start_us":9970,"threads_started":1}
I20260806 10:58:34.032320 18867 consensus_queue.cc:237] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe048dd5caa4a7caebc166a255fd3f6" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33163 } } peers { permanent_uuid: "916c04b37bdf475bac1fca90f1deb077" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 39443 } } peers { permanent_uuid: "2af87fb5d2d54b188fa72bfc1146191a" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 35365 } }
I20260806 10:58:34.033478 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ec51dc39e84036a0e5993a9a6cf005" candidate_uuid: "2af87fb5d2d54b188fa72bfc1146191a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efe048dd5caa4a7caebc166a255fd3f6" is_pre_election: true
I20260806 10:58:34.034500 18451 leader_election.cc:302] T 12ec51dc39e84036a0e5993a9a6cf005 P 2af87fb5d2d54b188fa72bfc1146191a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2af87fb5d2d54b188fa72bfc1146191a; no voters: 916c04b37bdf475bac1fca90f1deb077, efe048dd5caa4a7caebc166a255fd3f6
I20260806 10:58:34.034991 18866 raft_consensus.cc:2726] T 12ec51dc39e84036a0e5993a9a6cf005 P 2af87fb5d2d54b188fa72bfc1146191a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 10:58:34.037427 18374 catalog_manager.cc:5495] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to efe048dd5caa4a7caebc166a255fd3f6 (127.30.218.2). New cstate: current_term: 1 leader_uuid: "efe048dd5caa4a7caebc166a255fd3f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe048dd5caa4a7caebc166a255fd3f6" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 33163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "916c04b37bdf475bac1fca90f1deb077" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 39443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2af87fb5d2d54b188fa72bfc1146191a" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 35365 } health_report { overall_health: UNKNOWN } } }
I20260806 10:58:34.087188 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:34.081907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51450 (local address 127.30.218.2:33163)
0806 10:58:34.082071 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:34.082076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:34.082094 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:58:34.082203 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:34.082208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:34.082289 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:34.082419 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:58:34.082820 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:34.083607 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:34.085014 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:34.085484 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:34.085579 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:34.086134 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:58:34.086653 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:58:34.086798 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:34.086909 (+ 111us) server_negotiation.cc:299] Negotiation successful
0806 10:58:34.086949 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 10:58:34.096103 18794 raft_consensus.cc:1261] T 12ec51dc39e84036a0e5993a9a6cf005 P 916c04b37bdf475bac1fca90f1deb077 [term 1 FOLLOWER]: Refusing update from remote peer efe048dd5caa4a7caebc166a255fd3f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:58:34.096156 18523 raft_consensus.cc:1261] T 12ec51dc39e84036a0e5993a9a6cf005 P 2af87fb5d2d54b188fa72bfc1146191a [term 1 FOLLOWER]: Refusing update from remote peer efe048dd5caa4a7caebc166a255fd3f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:58:34.097008 18867 consensus_queue.cc:1035] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "916c04b37bdf475bac1fca90f1deb077" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 39443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:58:34.097411 18873 consensus_queue.cc:1035] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2af87fb5d2d54b188fa72bfc1146191a" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 35365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:58:34.108389 18877 mvcc.cc:205] Tried to move back new op lower bound from 7315512992126296064 to 7315512991880998912. Current Snapshot: MvccSnapshot[applied={T|T < 7315512992126296064}]
I20260806 10:58:34.110219 18878 mvcc.cc:205] Tried to move back new op lower bound from 7315512992126296064 to 7315512991880998912. Current Snapshot: MvccSnapshot[applied={T|T < 7315512992126296064}]
I20260806 10:58:34.122828 18375 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:58:34.123189 18375 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:58:34.127539 18375 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 742e496f302941e98dd57a135d923230: Sending DeleteTablet for 3 replicas of tablet 12ec51dc39e84036a0e5993a9a6cf005
I20260806 10:58:34.128957 18638 tablet_service.cc:1514] Processing DeleteTablet for tablet 12ec51dc39e84036a0e5993a9a6cf005 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:58:34 UTC) from {username='slave'} at 127.0.0.1:51442
I20260806 10:58:34.129245 18773 tablet_service.cc:1514] Processing DeleteTablet for tablet 12ec51dc39e84036a0e5993a9a6cf005 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:58:34 UTC) from {username='slave'} at 127.0.0.1:41496
I20260806 10:58:34.130189 18886 tablet_replica.cc:330] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6: stopping tablet replica
I20260806 10:58:34.130609 18502 tablet_service.cc:1514] Processing DeleteTablet for tablet 12ec51dc39e84036a0e5993a9a6cf005 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:58:34 UTC) from {username='slave'} at 127.0.0.1:55378
I20260806 10:58:34.130697 18886 raft_consensus.cc:2229] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:58:34.131471 18886 raft_consensus.cc:2258] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:58:34.131680 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 18436
I20260806 10:58:34.140982 18886 ts_tablet_manager.cc:1905] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:58:34.142007 18888 tablet_replica.cc:330] T 12ec51dc39e84036a0e5993a9a6cf005 P 2af87fb5d2d54b188fa72bfc1146191a: stopping tablet replica
I20260806 10:58:34.143812 18888 raft_consensus.cc:2229] T 12ec51dc39e84036a0e5993a9a6cf005 P 2af87fb5d2d54b188fa72bfc1146191a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:58:34.144430 18888 raft_consensus.cc:2258] T 12ec51dc39e84036a0e5993a9a6cf005 P 2af87fb5d2d54b188fa72bfc1146191a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:58:34.145738 18887 tablet_replica.cc:330] T 12ec51dc39e84036a0e5993a9a6cf005 P 916c04b37bdf475bac1fca90f1deb077: stopping tablet replica
I20260806 10:58:34.146260 18887 raft_consensus.cc:2229] T 12ec51dc39e84036a0e5993a9a6cf005 P 916c04b37bdf475bac1fca90f1deb077 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:58:34.146839 18887 raft_consensus.cc:2258] T 12ec51dc39e84036a0e5993a9a6cf005 P 916c04b37bdf475bac1fca90f1deb077 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:58:34.149837 18886 ts_tablet_manager.cc:1918] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:58:34.150068 18886 log.cc:1196] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/12ec51dc39e84036a0e5993a9a6cf005
I20260806 10:58:34.150503 18888 ts_tablet_manager.cc:1905] T 12ec51dc39e84036a0e5993a9a6cf005 P 2af87fb5d2d54b188fa72bfc1146191a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:58:34.150682 18886 ts_tablet_manager.cc:1939] T 12ec51dc39e84036a0e5993a9a6cf005 P efe048dd5caa4a7caebc166a255fd3f6: Deleting consensus metadata
I20260806 10:58:34.152891 18360 catalog_manager.cc:4841] TS efe048dd5caa4a7caebc166a255fd3f6 (127.30.218.2:33163): tablet 12ec51dc39e84036a0e5993a9a6cf005 (table test-table [id=ecd78263bf7848d19870ea6971804fa5]) successfully deleted
I20260806 10:58:34.154094 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:34.132832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55380 (local address 127.30.218.1:35365)
0806 10:58:34.133014 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:34.133021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:34.143947 (+ 10926us) server_negotiation.cc:407] Connection header received
0806 10:58:34.144638 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:34.144645 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:34.144772 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:34.144850 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:34.145566 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:34.146722 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:34.148945 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:34.149635 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:34.149744 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:34.152194 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:34.152218 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:34.152236 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:34.153160 (+ 924us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:34.153254 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:34.153284 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:34.153393 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:34.153577 (+ 184us) server_negotiation.cc:299] Negotiation successful
0806 10:58:34.153705 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 10:58:34.154896 18565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:34.156078 18887 ts_tablet_manager.cc:1905] T 12ec51dc39e84036a0e5993a9a6cf005 P 916c04b37bdf475bac1fca90f1deb077: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:58:34.162807 18887 ts_tablet_manager.cc:1918] T 12ec51dc39e84036a0e5993a9a6cf005 P 916c04b37bdf475bac1fca90f1deb077: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:58:34.163040 18887 log.cc:1196] T 12ec51dc39e84036a0e5993a9a6cf005 P 916c04b37bdf475bac1fca90f1deb077: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/12ec51dc39e84036a0e5993a9a6cf005
I20260806 10:58:34.163533 18887 ts_tablet_manager.cc:1939] T 12ec51dc39e84036a0e5993a9a6cf005 P 916c04b37bdf475bac1fca90f1deb077: Deleting consensus metadata
I20260806 10:58:34.165285 18361 catalog_manager.cc:4841] TS 916c04b37bdf475bac1fca90f1deb077 (127.30.218.3:39443): tablet 12ec51dc39e84036a0e5993a9a6cf005 (table test-table [id=ecd78263bf7848d19870ea6971804fa5]) successfully deleted
I20260806 10:58:34.310082 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 18436
W20260806 10:58:34.329236 18362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.218.1:35365 (error 108)
I20260806 10:58:34.329365 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 18574
I20260806 10:58:34.331818 18892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:34.330939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:35365 (local address 127.0.0.1:55396)
0806 10:58:34.331497 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:34.331645 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:35365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":342,"threads_started":1}
W20260806 10:58:34.332121 18362 catalog_manager.cc:4584] TS 2af87fb5d2d54b188fa72bfc1146191a (127.30.218.1:35365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12ec51dc39e84036a0e5993a9a6cf005: Network error: Client connection negotiation failed: client connection to 127.30.218.1:35365: connect: Connection refused (error 111)
I20260806 10:58:34.340646 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:34.330529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51458 (local address 127.30.218.2:33163)
0806 10:58:34.330722 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:34.330727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:34.330747 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:58:34.330804 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:34.330808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:34.330911 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:34.331026 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:34.331522 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:34.332695 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:34.334096 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:34.334710 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:34.334856 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:34.339094 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:34.339105 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:34.339121 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:34.339795 (+ 674us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:34.339854 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:34.339881 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:34.340011 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:34.340172 (+ 161us) server_negotiation.cc:299] Negotiation successful
0806 10:58:34.340300 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 10:58:34.341434 18702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:34.385946 18892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:34.385382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:35365 (local address 127.0.0.1:55412)
0806 10:58:34.385636 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:34.385743 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:35365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260806 10:58:34.387251 18892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:34.386810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:35365 (local address 127.0.0.1:55418)
0806 10:58:34.386988 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:34.387081 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:35365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 10:58:34.447644 18892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:34.447158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:35365 (local address 127.0.0.1:55430)
0806 10:58:34.447378 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:34.447474 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:35365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 10:58:34.448664 18892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:34.448326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:35365 (local address 127.0.0.1:55446)
0806 10:58:34.448440 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:34.448511 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:35365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260806 10:58:34.469888 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 18574
I20260806 10:58:34.488220 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 18710
I20260806 10:58:34.494731 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:34.488989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41508 (local address 127.30.218.3:39443)
0806 10:58:34.489184 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:34.489198 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:34.489218 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:58:34.489354 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:34.489358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:34.489476 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:34.489567 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:34.489884 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:34.490575 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:34.491823 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:34.492275 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:34.492370 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:34.493413 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:34.493423 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:34.493437 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:34.494071 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:34.494129 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:34.494150 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:34.494249 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:34.494346 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 10:58:34.494434 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 10:58:34.495428 18836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:34.529539 18892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:34.528975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:35365 (local address 127.0.0.1:55452)
0806 10:58:34.529227 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:34.529323 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:35365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260806 10:58:34.530748 18892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:34.530308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:35365 (local address 127.0.0.1:55468)
0806 10:58:34.530519 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:34.530592 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:35365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260806 10:58:34.586565 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 18710
I20260806 10:58:34.603777 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 18345
I20260806 10:58:34.604899 18406 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:34.718065 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 18345
2026-08-06T10:58:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3068 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:58:34 dist-test-slave-49h3 krb5kdc[18898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:58:34 dist-test-slave-49h3 krb5kdc[18898](info): set up 2 sockets
Aug 06 10:58:34 dist-test-slave-49h3 krb5kdc[18898](info): commencing operation
krb5kdc: starting...
W20260806 10:58:35.802518 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:58:36 dist-test-slave-49h3 krb5kdc[18898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:58:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:58:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:58:36.149185 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:40383
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:34769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:40383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:58:36.504232 18911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:36.504544 18911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:36.504643 18911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:36.512071 18911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:58:36.512209 18911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:36.512267 18911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:36.512323 18911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:58:36.512374 18911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:58:36.522437 18911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:34769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:40383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:40383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:36.523808 18911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:36.525072 18911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:36.536931 18916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:36.537010 18919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:36.536968 18917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:36.537962 18911 server_base.cc:1034] running on GCE node
I20260806 10:58:36.538813 18911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:36.540318 18911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:36.541549 18911 hybrid_clock.cc:648] HybridClock initialized: now 1786013916541493 us; error 45 us; skew 500 ppm
I20260806 10:58:36.541927 18911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:58:36 dist-test-slave-49h3 krb5kdc[18898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013916, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:58:36.550179 18911 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:58:36.551987 18911 webserver.cc:460] Webserver started at http://127.30.218.62:34933/ using document root <none> and password file <none>
I20260806 10:58:36.552757 18911 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:36.552875 18911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:36.553164 18911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:36.556409 18911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "75b00ee907c14c04ad589bda0a9d7a45"
format_stamp: "Formatted at 2026-08-06 10:58:36 on dist-test-slave-49h3"
server_key: "ff42c4826983bbf677f1f48175b7308c"
server_key_iv: "09a9b6a541fb9bef2011227189593dea"
server_key_version: "encryptionkey@0"
I20260806 10:58:36.557153 18911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "75b00ee907c14c04ad589bda0a9d7a45"
format_stamp: "Formatted at 2026-08-06 10:58:36 on dist-test-slave-49h3"
server_key: "ff42c4826983bbf677f1f48175b7308c"
server_key_iv: "09a9b6a541fb9bef2011227189593dea"
server_key_version: "encryptionkey@0"
I20260806 10:58:36.562839 18911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 10:58:36.566895 18926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:36.568387 18911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:58:36.568571 18911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "75b00ee907c14c04ad589bda0a9d7a45"
format_stamp: "Formatted at 2026-08-06 10:58:36 on dist-test-slave-49h3"
server_key: "ff42c4826983bbf677f1f48175b7308c"
server_key_iv: "09a9b6a541fb9bef2011227189593dea"
server_key_version: "encryptionkey@0"
I20260806 10:58:36.568775 18911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:36.580364 18911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:36.584236 18911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:36.584502 18911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:36.602905 18911 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:40383
I20260806 10:58:36.602923 18980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:40383 every 8 connection(s)
I20260806 10:58:36.604370 18911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:58:36.605350 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 18911
I20260806 10:58:36.605563 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:58:36.606195 31592 external_mini_cluster.cc:1428] Setting key d568eea843a991dc5ddbdeab5f9d1aa6
I20260806 10:58:36.609122 18981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 10:58:36 dist-test-slave-49h3 krb5kdc[18898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:58:36.622008 18981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75b00ee907c14c04ad589bda0a9d7a45: Bootstrap starting.
I20260806 10:58:36.625237 18981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75b00ee907c14c04ad589bda0a9d7a45: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:36.626605 18981 log.cc:824] T 00000000000000000000000000000000 P 75b00ee907c14c04ad589bda0a9d7a45: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:36.629580 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:36.608140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.30.218.62:40383)
0806 10:58:36.608944 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:36.608955 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:36.608997 (+ 42us) server_negotiation.cc:407] Connection header received
0806 10:58:36.610727 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:36.610748 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:36.611105 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:36.611433 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:36.613271 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:36.614666 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:36.616462 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:36.617099 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:36.617220 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:36.622240 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:36.622249 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:36.622263 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:36.622301 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:36.625203 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:36.626141 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:36.626148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:36.626155 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:36.626331 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:36.626771 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:36.626775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:36.626778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:36.627460 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:36.627782 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:36.628123 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 10:58:36.628546 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":367,"threads_started":1}
I20260806 10:58:36.629889 18981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75b00ee907c14c04ad589bda0a9d7a45: No bootstrap required, opened a new log
I20260806 10:58:36.637432 18981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75b00ee907c14c04ad589bda0a9d7a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b00ee907c14c04ad589bda0a9d7a45" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40383 } }
I20260806 10:58:36.637920 18981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75b00ee907c14c04ad589bda0a9d7a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:36.638015 18981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75b00ee907c14c04ad589bda0a9d7a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b00ee907c14c04ad589bda0a9d7a45, State: Initialized, Role: FOLLOWER
I20260806 10:58:36.638587 18981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75b00ee907c14c04ad589bda0a9d7a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b00ee907c14c04ad589bda0a9d7a45" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40383 } }
I20260806 10:58:36.638808 18981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75b00ee907c14c04ad589bda0a9d7a45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:58:36.638936 18981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75b00ee907c14c04ad589bda0a9d7a45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:58:36.639092 18981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75b00ee907c14c04ad589bda0a9d7a45 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:36.641309 18981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75b00ee907c14c04ad589bda0a9d7a45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b00ee907c14c04ad589bda0a9d7a45" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40383 } }
I20260806 10:58:36.641863 18981 leader_election.cc:302] T 00000000000000000000000000000000 P 75b00ee907c14c04ad589bda0a9d7a45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75b00ee907c14c04ad589bda0a9d7a45; no voters:
I20260806 10:58:36.642423 18981 leader_election.cc:288] T 00000000000000000000000000000000 P 75b00ee907c14c04ad589bda0a9d7a45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:58:36.642552 18986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75b00ee907c14c04ad589bda0a9d7a45 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:58:36.643196 18986 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75b00ee907c14c04ad589bda0a9d7a45 [term 1 LEADER]: Becoming Leader. State: Replica: 75b00ee907c14c04ad589bda0a9d7a45, State: Running, Role: LEADER
I20260806 10:58:36.643743 18986 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75b00ee907c14c04ad589bda0a9d7a45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b00ee907c14c04ad589bda0a9d7a45" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40383 } }
I20260806 10:58:36.644480 18981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75b00ee907c14c04ad589bda0a9d7a45 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:58:36.646615 18987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75b00ee907c14c04ad589bda0a9d7a45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75b00ee907c14c04ad589bda0a9d7a45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b00ee907c14c04ad589bda0a9d7a45" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40383 } } }
I20260806 10:58:36.646646 18988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75b00ee907c14c04ad589bda0a9d7a45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75b00ee907c14c04ad589bda0a9d7a45. Latest consensus state: current_term: 1 leader_uuid: "75b00ee907c14c04ad589bda0a9d7a45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b00ee907c14c04ad589bda0a9d7a45" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40383 } } }
I20260806 10:58:36.647452 18988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75b00ee907c14c04ad589bda0a9d7a45 [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:36.647261 18987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75b00ee907c14c04ad589bda0a9d7a45 [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:36.650053 18993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:58:36.655288 18993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:58:36.664188 18993 catalog_manager.cc:1347] Generated new cluster ID: 800b25ce9af349ea99a15a735cde6cd8
I20260806 10:58:36.664325 18993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:58:36.684528 18993 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:58:36.686578 18993 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:58:36.699857 18993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75b00ee907c14c04ad589bda0a9d7a45: Generated new TSK 0
I20260806 10:58:36.700838 18993 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
Entry for principal kudu/127.30.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab.
I20260806 10:58:36.793037 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:40383
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:34769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:58:37.146365 19007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:37.146653 19007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:37.146775 19007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:37.154304 19007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:37.154431 19007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:37.154585 19007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:58:37.164829 19007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:34769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.1.keytab
--principal=kudu/127.30.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:40383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:37.166307 19007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:37.167722 19007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:37.180812 19012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:37.180693 19015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:37.181226 19013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:37.182297 19007 server_base.cc:1034] running on GCE node
I20260806 10:58:37.182940 19007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:37.183948 19007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:37.185161 19007 hybrid_clock.cc:648] HybridClock initialized: now 1786013917185096 us; error 59 us; skew 500 ppm
I20260806 10:58:37.185547 19007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:58:37 dist-test-slave-49h3 krb5kdc[18898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013917, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:58:37.188783 19007 init.cc:377] Logged in from keytab as kudu/127.30.218.1@KRBTEST.COM (short username kudu)
I20260806 10:58:37.190418 19007 webserver.cc:460] Webserver started at http://127.30.218.1:44105/ using document root <none> and password file <none>
I20260806 10:58:37.191138 19007 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:37.191283 19007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:37.191612 19007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:37.194859 19007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "1c95860436ba46bab2d945dacf32270e"
format_stamp: "Formatted at 2026-08-06 10:58:37 on dist-test-slave-49h3"
server_key: "7be820283ec3e30cbc68d4e898cbdafa"
server_key_iv: "1326eadd1541415143aa5af262f452a6"
server_key_version: "encryptionkey@0"
I20260806 10:58:37.195705 19007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "1c95860436ba46bab2d945dacf32270e"
format_stamp: "Formatted at 2026-08-06 10:58:37 on dist-test-slave-49h3"
server_key: "7be820283ec3e30cbc68d4e898cbdafa"
server_key_iv: "1326eadd1541415143aa5af262f452a6"
server_key_version: "encryptionkey@0"
I20260806 10:58:37.201741 19007 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 10:58:37.205801 19022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:37.207391 19007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:58:37.207574 19007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "1c95860436ba46bab2d945dacf32270e"
format_stamp: "Formatted at 2026-08-06 10:58:37 on dist-test-slave-49h3"
server_key: "7be820283ec3e30cbc68d4e898cbdafa"
server_key_iv: "1326eadd1541415143aa5af262f452a6"
server_key_version: "encryptionkey@0"
I20260806 10:58:37.207796 19007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:37.226068 19007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:37.230480 19007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:37.230791 19007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:37.231873 19007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:58:37.233369 19007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:58:37.233469 19007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:37.233596 19007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:58:37.233647 19007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:37.265600 19007 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:41219
I20260806 10:58:37.265655 19137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:41219 every 8 connection(s)
I20260806 10:58:37.266976 19007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:58:37.275866 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 19007
I20260806 10:58:37.276034 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:58:37.276461 31592 external_mini_cluster.cc:1428] Setting key 51c20a0214e9c9269642fec2b2e1f0d0
Aug 06 10:58:37 dist-test-slave-49h3 krb5kdc[18898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013917, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.1@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:58:37.286855 19141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:37.269556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:38569 (local address 127.30.218.62:40383)
0806 10:58:37.269998 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:37.270003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:37.270990 (+ 987us) server_negotiation.cc:407] Connection header received
0806 10:58:37.272869 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:37.272875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:37.272992 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:37.273127 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:37.274716 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:37.275429 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:37.277132 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:37.277688 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:37.277816 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:37.281303 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:37.281309 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:37.281312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:37.281350 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:37.283800 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:37.284513 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:37.284519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:37.284523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:37.284649 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:37.285067 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:37.285073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:37.285076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:37.285350 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:37.285521 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:37.286245 (+ 724us) server_negotiation.cc:299] Negotiation successful
0806 10:58:37.286483 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":207,"threads_started":1}
I20260806 10:58:37.287811 19142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:37.270098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:40383 (local address 127.30.218.1:38569)
0806 10:58:37.270829 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:37.270869 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:37.272602 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:37.273348 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:37.273357 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:37.273956 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:58:37.274519 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:37.274537 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:37.275599 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:37.275604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:37.276984 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:37.276998 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:37.277899 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:37.277904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:37.278145 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:37.278858 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:58:37.278886 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:58:37.281053 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:58:37.283938 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:58:37.283947 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:58:37.283966 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:58:37.284397 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:58:37.284765 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:58:37.284770 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:58:37.284773 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:58:37.284959 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:58:37.285500 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:58:37.285508 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:58:37.285857 (+ 349us) client_negotiation.cc:769] Sending connection context
0806 10:58:37.286123 (+ 266us) client_negotiation.cc:240] Negotiation successful
0806 10:58:37.286672 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":275,"threads_started":1}
I20260806 10:58:37.289372 19138 heartbeater.cc:347] Connected to a master server at 127.30.218.62:40383
I20260806 10:58:37.289740 19138 heartbeater.cc:464] Registering TS with master...
I20260806 10:58:37.290611 19138 heartbeater.cc:511] Master 127.30.218.62:40383 requested a full tablet report, sending...
I20260806 10:58:37.292930 18944 ts_manager.cc:194] Registered new tserver with Master: 1c95860436ba46bab2d945dacf32270e (127.30.218.1:41219)
I20260806 10:58:37.294457 18944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.1@KRBTEST.COM'} at 127.30.218.1:38569
WARNING: no policy specified for kudu/127.30.218.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
Entry for principal kudu/127.30.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab.
I20260806 10:58:37.346832 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:40383
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:34769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:58:37.701524 19145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:37.701872 19145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:37.702008 19145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:37.709781 19145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:37.709939 19145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:37.710093 19145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:58:37.720314 19145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:34769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.2.keytab
--principal=kudu/127.30.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:40383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:37.721771 19145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:37.723151 19145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:37.736236 19150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:37.736167 19151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:37.736210 19153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:37.737052 19145 server_base.cc:1034] running on GCE node
I20260806 10:58:37.737699 19145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:37.738643 19145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:37.739845 19145 hybrid_clock.cc:648] HybridClock initialized: now 1786013917739793 us; error 47 us; skew 500 ppm
I20260806 10:58:37.740145 19145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:58:37 dist-test-slave-49h3 krb5kdc[18898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013917, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:58:37.743340 19145 init.cc:377] Logged in from keytab as kudu/127.30.218.2@KRBTEST.COM (short username kudu)
I20260806 10:58:37.744880 19145 webserver.cc:460] Webserver started at http://127.30.218.2:34455/ using document root <none> and password file <none>
I20260806 10:58:37.745639 19145 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:37.745764 19145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:37.746083 19145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:37.749038 19145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "02546e2b2b7443c1b44a97837a41836a"
format_stamp: "Formatted at 2026-08-06 10:58:37 on dist-test-slave-49h3"
server_key: "788ded3130d8ef08664561480c61cfbf"
server_key_iv: "93572ee437b62182c63b5cbe290f85b7"
server_key_version: "encryptionkey@0"
I20260806 10:58:37.749861 19145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "02546e2b2b7443c1b44a97837a41836a"
format_stamp: "Formatted at 2026-08-06 10:58:37 on dist-test-slave-49h3"
server_key: "788ded3130d8ef08664561480c61cfbf"
server_key_iv: "93572ee437b62182c63b5cbe290f85b7"
server_key_version: "encryptionkey@0"
I20260806 10:58:37.755019 19145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 10:58:37.758491 19160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:37.759810 19145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:58:37.759959 19145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "02546e2b2b7443c1b44a97837a41836a"
format_stamp: "Formatted at 2026-08-06 10:58:37 on dist-test-slave-49h3"
server_key: "788ded3130d8ef08664561480c61cfbf"
server_key_iv: "93572ee437b62182c63b5cbe290f85b7"
server_key_version: "encryptionkey@0"
I20260806 10:58:37.760144 19145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:37.775318 19145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:37.779562 19145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:37.779844 19145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:37.780812 19145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:58:37.782245 19145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:58:37.782339 19145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:37.782465 19145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:58:37.782526 19145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:37.814043 19145 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:42887
I20260806 10:58:37.814092 19275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:42887 every 8 connection(s)
I20260806 10:58:37.815400 19145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:58:37.821156 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 19145
I20260806 10:58:37.821355 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:58:37.822002 31592 external_mini_cluster.cc:1428] Setting key 52a7c71b1af2c5224c6f4b62264be595
Aug 06 10:58:37 dist-test-slave-49h3 krb5kdc[18898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013917, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.2@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:58:37.836608 19279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:37.817974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:41171 (local address 127.30.218.62:40383)
0806 10:58:37.818416 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:37.818421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:37.819275 (+ 854us) server_negotiation.cc:407] Connection header received
0806 10:58:37.821091 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:37.821098 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:37.821180 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:37.821292 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:37.823330 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:37.824346 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:37.826477 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:37.826888 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:37.826979 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:37.830962 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:37.830967 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:37.830970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:37.831003 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:37.833555 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:37.834395 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:37.834399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:37.834401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:37.834527 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:37.834971 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:37.834974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:37.834975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:37.835230 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:37.835341 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:37.836138 (+ 797us) server_negotiation.cc:299] Negotiation successful
0806 10:58:37.836316 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":236,"threads_started":1}
I20260806 10:58:37.838040 19280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:37.818395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:40383 (local address 127.30.218.2:41171)
0806 10:58:37.819099 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:37.819142 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:37.820817 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:37.821617 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:37.821630 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:37.822406 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:58:37.823150 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:37.823171 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:37.824516 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:37.824522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:37.826349 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:37.826366 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:37.827101 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:37.827109 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:37.827379 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:37.828052 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:58:37.828078 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:58:37.830705 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:58:37.833695 (+ 2990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:58:37.833705 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:58:37.833723 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:58:37.834291 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:58:37.834621 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:58:37.834625 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:58:37.834628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:58:37.834877 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:58:37.835343 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:58:37.835351 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:58:37.835792 (+ 441us) client_negotiation.cc:769] Sending connection context
0806 10:58:37.836076 (+ 284us) client_negotiation.cc:240] Negotiation successful
0806 10:58:37.836622 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":270,"threads_started":1}
I20260806 10:58:37.839764 19276 heartbeater.cc:347] Connected to a master server at 127.30.218.62:40383
I20260806 10:58:37.840080 19276 heartbeater.cc:464] Registering TS with master...
I20260806 10:58:37.840838 19276 heartbeater.cc:511] Master 127.30.218.62:40383 requested a full tablet report, sending...
I20260806 10:58:37.842448 18944 ts_manager.cc:194] Registered new tserver with Master: 02546e2b2b7443c1b44a97837a41836a (127.30.218.2:42887)
I20260806 10:58:37.843317 18944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.2@KRBTEST.COM'} at 127.30.218.2:41171
WARNING: no policy specified for kudu/127.30.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
Entry for principal kudu/127.30.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab.
I20260806 10:58:37.897154 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:40383
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:34769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:58:38.239939 19283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:38.240247 19283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:38.240398 19283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:38.247970 19283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:38.248095 19283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:38.248251 19283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:58:38.258296 19283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:34769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.3.keytab
--principal=kudu/127.30.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:40383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:38.259646 19283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:38.260962 19283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:38.274986 19288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:38.275059 19291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:38.275030 19289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:38.275965 19283 server_base.cc:1034] running on GCE node
I20260806 10:58:38.276638 19283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:38.277745 19283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:38.278970 19283 hybrid_clock.cc:648] HybridClock initialized: now 1786013918278906 us; error 51 us; skew 500 ppm
I20260806 10:58:38.279320 19283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:58:38 dist-test-slave-49h3 krb5kdc[18898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013918, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:58:38.282728 19283 init.cc:377] Logged in from keytab as kudu/127.30.218.3@KRBTEST.COM (short username kudu)
I20260806 10:58:38.284457 19283 webserver.cc:460] Webserver started at http://127.30.218.3:46775/ using document root <none> and password file <none>
I20260806 10:58:38.285183 19283 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:38.285306 19283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:38.285661 19283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:38.289039 19283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "1da5ad6830b34ccaac7ea8194f8b850c"
format_stamp: "Formatted at 2026-08-06 10:58:38 on dist-test-slave-49h3"
server_key: "759301a727f81a0d3b2bb3e33aa195cc"
server_key_iv: "1b91f6b95fa2a4f6275d4b96c9798ffd"
server_key_version: "encryptionkey@0"
I20260806 10:58:38.289947 19283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "1da5ad6830b34ccaac7ea8194f8b850c"
format_stamp: "Formatted at 2026-08-06 10:58:38 on dist-test-slave-49h3"
server_key: "759301a727f81a0d3b2bb3e33aa195cc"
server_key_iv: "1b91f6b95fa2a4f6275d4b96c9798ffd"
server_key_version: "encryptionkey@0"
I20260806 10:58:38.295825 19283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:58:38.297780 19138 heartbeater.cc:503] Master 127.30.218.62:40383 was elected leader, sending a full tablet report...
I20260806 10:58:38.300094 19298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:38.301679 19283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:58:38.301857 19283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "1da5ad6830b34ccaac7ea8194f8b850c"
format_stamp: "Formatted at 2026-08-06 10:58:38 on dist-test-slave-49h3"
server_key: "759301a727f81a0d3b2bb3e33aa195cc"
server_key_iv: "1b91f6b95fa2a4f6275d4b96c9798ffd"
server_key_version: "encryptionkey@0"
I20260806 10:58:38.302070 19283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:38.322311 19283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:38.326749 19283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:38.327026 19283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:38.328063 19283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:58:38.329707 19283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:58:38.329792 19283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:38.329898 19283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:58:38.329943 19283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:38.362401 19283 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:37739
I20260806 10:58:38.362438 19412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:37739 every 8 connection(s)
I20260806 10:58:38.363863 19283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:58:38.370326 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 19283
I20260806 10:58:38.370517 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:58:38.370959 31592 external_mini_cluster.cc:1428] Setting key 5fb92b8d0dd23027110199c9108bbfe6
Aug 06 10:58:38 dist-test-slave-49h3 krb5kdc[18898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013918, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.3@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:58:38.385605 19415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:38.366702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:40411 (local address 127.30.218.62:40383)
0806 10:58:38.367183 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:38.367189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:38.368091 (+ 902us) server_negotiation.cc:407] Connection header received
0806 10:58:38.370108 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:38.370115 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:38.370200 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:38.370308 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:38.372217 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.373088 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.374802 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.375234 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.375350 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:38.379244 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:38.379250 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:38.379253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:38.379292 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:38.381925 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:38.382827 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:38.382832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:38.382835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:38.382975 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:38.383563 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:38.383567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:38.383569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:38.383884 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:38.384044 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:38.384870 (+ 826us) server_negotiation.cc:299] Negotiation successful
0806 10:58:38.385143 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":242,"threads_started":1}
I20260806 10:58:38.386833 19416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:38.367157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:40383 (local address 127.30.218.3:40411)
0806 10:58:38.367911 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:38.367954 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:38.369782 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:38.370615 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:38.370628 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:38.371268 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:58:38.372015 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:38.372035 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.373251 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.373255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:38.374638 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:38.374653 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.375480 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.375488 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:38.375730 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:38.376417 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:58:38.376448 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:58:38.378969 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:58:38.382111 (+ 3142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:58:38.382122 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:58:38.382140 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:58:38.382656 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:58:38.383097 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:58:38.383102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:58:38.383106 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:58:38.383431 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:58:38.384028 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:58:38.384037 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:58:38.384447 (+ 410us) client_negotiation.cc:769] Sending connection context
0806 10:58:38.384720 (+ 273us) client_negotiation.cc:240] Negotiation successful
0806 10:58:38.385332 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":276,"threads_started":1}
I20260806 10:58:38.388237 19413 heartbeater.cc:347] Connected to a master server at 127.30.218.62:40383
I20260806 10:58:38.388577 19413 heartbeater.cc:464] Registering TS with master...
I20260806 10:58:38.389384 19413 heartbeater.cc:511] Master 127.30.218.62:40383 requested a full tablet report, sending...
I20260806 10:58:38.391106 18944 ts_manager.cc:194] Registered new tserver with Master: 1da5ad6830b34ccaac7ea8194f8b850c (127.30.218.3:37739)
I20260806 10:58:38.392031 18944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.218.3@KRBTEST.COM'} at 127.30.218.3:40411
I20260806 10:58:38.399408 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:58:38.416841 19415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:38.403791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40266 (local address 127.30.218.62:40383)
0806 10:58:38.403983 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:38.403990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:38.404411 (+ 421us) server_negotiation.cc:407] Connection header received
0806 10:58:38.404603 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:38.404608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:38.404683 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:38.404784 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:38.406427 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.407257 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.408557 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.408971 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.409091 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:38.411715 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:38.411720 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:38.411724 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:38.411766 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:38.414394 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:38.415088 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:38.415092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:38.415094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:38.415222 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:38.415549 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:38.415553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:38.415554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:38.415897 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:38.416087 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:38.416349 (+ 262us) server_negotiation.cc:299] Negotiation successful
0806 10:58:38.416544 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 10:58:38.420971 18944 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:58:38.428207 18944 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:58:38.465044 19432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:38.442478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34886 (local address 127.30.218.1:41219)
0806 10:58:38.452782 (+ 10304us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:38.452792 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:38.455882 (+ 3090us) server_negotiation.cc:407] Connection header received
0806 10:58:38.456526 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:38.456535 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:38.456832 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:38.457015 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:38.458196 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.459386 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.461954 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.463334 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.463477 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:38.463734 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:38.464464 (+ 730us) server_negotiation.cc:299] Negotiation successful
0806 10:58:38.464664 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10108,"thread_start_us":9921,"threads_started":1}
I20260806 10:58:38.465417 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:38.441120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:37739 (local address 127.0.0.1:32818)
0806 10:58:38.442091 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:38.442142 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:38.442320 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:38.457856 (+ 15536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:38.457861 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:38.457890 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:38.458356 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:38.458371 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.460308 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.460314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:38.462636 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:38.462650 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.464467 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.464473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:38.464645 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:38.464676 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 10:58:38.464729 (+ 53us) client_negotiation.cc:240] Negotiation successful
0806 10:58:38.464954 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":6,"thread_start_us":580,"threads_started":1}
I20260806 10:58:38.465873 19431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:38.441792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:41219 (local address 127.0.0.1:34886)
0806 10:58:38.455740 (+ 13948us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:38.455760 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:38.455915 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:38.457311 (+ 1396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:38.457318 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:38.457360 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:38.458010 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:38.458020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.459563 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.459568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:38.461775 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:38.461793 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.463898 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.463905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:38.464159 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:38.464234 (+ 75us) client_negotiation.cc:769] Sending connection context
0806 10:58:38.465305 (+ 1071us) client_negotiation.cc:240] Negotiation successful
0806 10:58:38.465492 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13813,"mutex_wait_us":57,"spinlock_wait_cycles":1792,"thread_start_us":14246,"threads_started":1}
I20260806 10:58:38.465885 19433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:38.442826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42516 (local address 127.30.218.2:42887)
0806 10:58:38.455618 (+ 12792us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:38.455626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:38.456558 (+ 932us) server_negotiation.cc:407] Connection header received
0806 10:58:38.456651 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:38.456658 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:38.456835 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:38.457018 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:38.458227 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.459732 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.462544 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.464185 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.464349 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:38.464537 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:38.465103 (+ 566us) server_negotiation.cc:299] Negotiation successful
0806 10:58:38.465605 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12625,"thread_start_us":12808,"threads_started":1}
I20260806 10:58:38.466593 19429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:38.441382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32818 (local address 127.30.218.3:37739)
0806 10:58:38.457335 (+ 15953us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:38.457343 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:38.457373 (+ 30us) server_negotiation.cc:407] Connection header received
0806 10:58:38.457474 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:38.457482 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:38.457657 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:38.457835 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:38.458579 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.460127 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.462804 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.464251 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.465567 (+ 1316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:38.465826 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:38.465905 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 10:58:38.466324 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15761,"thread_start_us":15765,"threads_started":1}
I20260806 10:58:38.466600 19430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:38.442474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:42887 (local address 127.0.0.1:42516)
0806 10:58:38.456300 (+ 13826us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:38.456316 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:38.456435 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:38.457310 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:38.457319 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:38.457369 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:38.457954 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:38.457975 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.459930 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.459936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:38.462350 (+ 2414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:38.462366 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.464690 (+ 2324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.464696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:38.464891 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:38.464923 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 10:58:38.465028 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 10:58:38.465177 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13702,"mutex_wait_us":60,"spinlock_wait_cycles":4736,"thread_start_us":13538,"threads_started":1}
I20260806 10:58:38.470351 19209 tablet_service.cc:1467] Processing CreateTablet for tablet 0893dfa0e92a40fc96b91236b03ae62c (DEFAULT_TABLE table=test-table [id=dc1f237e3e97400e9eb2881738427378]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:58:38.470350 19347 tablet_service.cc:1467] Processing CreateTablet for tablet 0893dfa0e92a40fc96b91236b03ae62c (DEFAULT_TABLE table=test-table [id=dc1f237e3e97400e9eb2881738427378]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:58:38.470350 19072 tablet_service.cc:1467] Processing CreateTablet for tablet 0893dfa0e92a40fc96b91236b03ae62c (DEFAULT_TABLE table=test-table [id=dc1f237e3e97400e9eb2881738427378]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:58:38.471760 19209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0893dfa0e92a40fc96b91236b03ae62c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:38.471817 19347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0893dfa0e92a40fc96b91236b03ae62c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:38.471820 19072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0893dfa0e92a40fc96b91236b03ae62c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:38.484972 19434 tablet_bootstrap.cc:492] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a: Bootstrap starting.
I20260806 10:58:38.488349 19434 tablet_bootstrap.cc:654] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:38.490044 19434 log.cc:824] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:38.492681 19435 tablet_bootstrap.cc:492] T 0893dfa0e92a40fc96b91236b03ae62c P 1c95860436ba46bab2d945dacf32270e: Bootstrap starting.
I20260806 10:58:38.493229 19434 tablet_bootstrap.cc:492] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a: No bootstrap required, opened a new log
I20260806 10:58:38.493546 19434 ts_tablet_manager.cc:1397] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 10:58:38.493615 19436 tablet_bootstrap.cc:492] T 0893dfa0e92a40fc96b91236b03ae62c P 1da5ad6830b34ccaac7ea8194f8b850c: Bootstrap starting.
I20260806 10:58:38.496518 19435 tablet_bootstrap.cc:654] T 0893dfa0e92a40fc96b91236b03ae62c P 1c95860436ba46bab2d945dacf32270e: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:38.496984 19436 tablet_bootstrap.cc:654] T 0893dfa0e92a40fc96b91236b03ae62c P 1da5ad6830b34ccaac7ea8194f8b850c: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:38.498020 19435 log.cc:824] T 0893dfa0e92a40fc96b91236b03ae62c P 1c95860436ba46bab2d945dacf32270e: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:38.498199 19436 log.cc:824] T 0893dfa0e92a40fc96b91236b03ae62c P 1da5ad6830b34ccaac7ea8194f8b850c: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:38.500900 19435 tablet_bootstrap.cc:492] T 0893dfa0e92a40fc96b91236b03ae62c P 1c95860436ba46bab2d945dacf32270e: No bootstrap required, opened a new log
I20260806 10:58:38.501174 19435 ts_tablet_manager.cc:1397] T 0893dfa0e92a40fc96b91236b03ae62c P 1c95860436ba46bab2d945dacf32270e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260806 10:58:38.501212 19436 tablet_bootstrap.cc:492] T 0893dfa0e92a40fc96b91236b03ae62c P 1da5ad6830b34ccaac7ea8194f8b850c: No bootstrap required, opened a new log
I20260806 10:58:38.501503 19436 ts_tablet_manager.cc:1397] T 0893dfa0e92a40fc96b91236b03ae62c P 1da5ad6830b34ccaac7ea8194f8b850c: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20260806 10:58:38.502264 19434 raft_consensus.cc:348] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da5ad6830b34ccaac7ea8194f8b850c" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 37739 } } peers { permanent_uuid: "02546e2b2b7443c1b44a97837a41836a" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42887 } } peers { permanent_uuid: "1c95860436ba46bab2d945dacf32270e" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 41219 } }
I20260806 10:58:38.502748 19434 raft_consensus.cc:374] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:38.502866 19434 raft_consensus.cc:729] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02546e2b2b7443c1b44a97837a41836a, State: Initialized, Role: FOLLOWER
I20260806 10:58:38.507803 19434 consensus_queue.cc:260] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da5ad6830b34ccaac7ea8194f8b850c" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 37739 } } peers { permanent_uuid: "02546e2b2b7443c1b44a97837a41836a" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42887 } } peers { permanent_uuid: "1c95860436ba46bab2d945dacf32270e" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 41219 } }
I20260806 10:58:38.510310 19276 heartbeater.cc:503] Master 127.30.218.62:40383 was elected leader, sending a full tablet report...
I20260806 10:58:38.511111 19434 ts_tablet_manager.cc:1428] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a: Time spent starting tablet: real 0.017s user 0.013s sys 0.000s
I20260806 10:58:38.511080 19435 raft_consensus.cc:348] T 0893dfa0e92a40fc96b91236b03ae62c P 1c95860436ba46bab2d945dacf32270e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da5ad6830b34ccaac7ea8194f8b850c" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 37739 } } peers { permanent_uuid: "02546e2b2b7443c1b44a97837a41836a" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42887 } } peers { permanent_uuid: "1c95860436ba46bab2d945dacf32270e" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 41219 } }
I20260806 10:58:38.511674 19435 raft_consensus.cc:374] T 0893dfa0e92a40fc96b91236b03ae62c P 1c95860436ba46bab2d945dacf32270e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:38.511788 19435 raft_consensus.cc:729] T 0893dfa0e92a40fc96b91236b03ae62c P 1c95860436ba46bab2d945dacf32270e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c95860436ba46bab2d945dacf32270e, State: Initialized, Role: FOLLOWER
I20260806 10:58:38.512004 19436 raft_consensus.cc:348] T 0893dfa0e92a40fc96b91236b03ae62c P 1da5ad6830b34ccaac7ea8194f8b850c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da5ad6830b34ccaac7ea8194f8b850c" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 37739 } } peers { permanent_uuid: "02546e2b2b7443c1b44a97837a41836a" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42887 } } peers { permanent_uuid: "1c95860436ba46bab2d945dacf32270e" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 41219 } }
I20260806 10:58:38.512475 19436 raft_consensus.cc:374] T 0893dfa0e92a40fc96b91236b03ae62c P 1da5ad6830b34ccaac7ea8194f8b850c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:38.512439 19435 consensus_queue.cc:260] T 0893dfa0e92a40fc96b91236b03ae62c P 1c95860436ba46bab2d945dacf32270e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da5ad6830b34ccaac7ea8194f8b850c" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 37739 } } peers { permanent_uuid: "02546e2b2b7443c1b44a97837a41836a" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42887 } } peers { permanent_uuid: "1c95860436ba46bab2d945dacf32270e" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 41219 } }
I20260806 10:58:38.512588 19436 raft_consensus.cc:729] T 0893dfa0e92a40fc96b91236b03ae62c P 1da5ad6830b34ccaac7ea8194f8b850c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da5ad6830b34ccaac7ea8194f8b850c, State: Initialized, Role: FOLLOWER
I20260806 10:58:38.513175 19436 consensus_queue.cc:260] T 0893dfa0e92a40fc96b91236b03ae62c P 1da5ad6830b34ccaac7ea8194f8b850c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da5ad6830b34ccaac7ea8194f8b850c" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 37739 } } peers { permanent_uuid: "02546e2b2b7443c1b44a97837a41836a" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42887 } } peers { permanent_uuid: "1c95860436ba46bab2d945dacf32270e" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 41219 } }
I20260806 10:58:38.514431 19435 ts_tablet_manager.cc:1428] T 0893dfa0e92a40fc96b91236b03ae62c P 1c95860436ba46bab2d945dacf32270e: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260806 10:58:38.515537 19413 heartbeater.cc:503] Master 127.30.218.62:40383 was elected leader, sending a full tablet report...
I20260806 10:58:38.516191 19436 ts_tablet_manager.cc:1428] T 0893dfa0e92a40fc96b91236b03ae62c P 1da5ad6830b34ccaac7ea8194f8b850c: Time spent starting tablet: real 0.015s user 0.008s sys 0.005s
W20260806 10:58:38.519357 19140 tablet.cc:2369] T 0893dfa0e92a40fc96b91236b03ae62c P 1c95860436ba46bab2d945dacf32270e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:58:38.567476 19278 tablet.cc:2369] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 10:58:38.615716 19414 tablet.cc:2369] T 0893dfa0e92a40fc96b91236b03ae62c P 1da5ad6830b34ccaac7ea8194f8b850c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:58:38.819702 19440 raft_consensus.cc:482] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:58:38.820041 19440 raft_consensus.cc:504] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da5ad6830b34ccaac7ea8194f8b850c" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 37739 } } peers { permanent_uuid: "02546e2b2b7443c1b44a97837a41836a" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42887 } } peers { permanent_uuid: "1c95860436ba46bab2d945dacf32270e" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 41219 } }
I20260806 10:58:38.821983 19440 leader_election.cc:288] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da5ad6830b34ccaac7ea8194f8b850c (127.30.218.3:37739), 1c95860436ba46bab2d945dacf32270e (127.30.218.1:41219)
I20260806 10:58:38.832032 19429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:38.822730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:45135 (local address 127.30.218.3:37739)
0806 10:58:38.823146 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:38.823152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:38.824183 (+ 1031us) server_negotiation.cc:407] Connection header received
0806 10:58:38.824327 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:38.824333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:38.824409 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:38.824494 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:38.825200 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.826389 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.829430 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.830735 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.830911 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:38.830995 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:38.831576 (+ 581us) server_negotiation.cc:299] Negotiation successful
0806 10:58:38.831739 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288}
I20260806 10:58:38.832036 19432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:38.822581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:36501 (local address 127.30.218.1:41219)
0806 10:58:38.822986 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:38.822993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:38.823787 (+ 794us) server_negotiation.cc:407] Connection header received
0806 10:58:38.824111 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:38.824116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:38.824204 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:38.824292 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:38.825369 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.826471 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.829423 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.830724 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.830910 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:38.830995 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:38.831576 (+ 581us) server_negotiation.cc:299] Negotiation successful
0806 10:58:38.831739 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236}
I20260806 10:58:38.832360 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:38.822798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:37739 (local address 127.30.218.2:45135)
0806 10:58:38.824057 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:38.824074 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:38.824209 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:38.824558 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:38.824562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:38.824586 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:38.824967 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:38.824976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.826574 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.826581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:38.829222 (+ 2641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:38.829243 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.831034 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.831042 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:38.831294 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:38.831386 (+ 92us) client_negotiation.cc:769] Sending connection context
0806 10:58:38.831493 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 10:58:38.831742 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1130,"mutex_wait_us":156,"thread_start_us":501,"threads_started":1}
I20260806 10:58:38.832383 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:38.822506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:41219 (local address 127.30.218.2:36501)
0806 10:58:38.823662 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:38.823682 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:38.823921 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:38.824671 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:38.824676 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:38.824718 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:38.825205 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:38.825217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.826613 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.826618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:38.829222 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:38.829243 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.831047 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.831053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:38.831294 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:38.831384 (+ 90us) client_negotiation.cc:769] Sending connection context
0806 10:58:38.831493 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 10:58:38.831742 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"mutex_wait_us":248,"thread_start_us":1200,"threads_started":1}
I20260806 10:58:38.833763 19368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0893dfa0e92a40fc96b91236b03ae62c" candidate_uuid: "02546e2b2b7443c1b44a97837a41836a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da5ad6830b34ccaac7ea8194f8b850c" is_pre_election: true
I20260806 10:58:38.833763 19093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0893dfa0e92a40fc96b91236b03ae62c" candidate_uuid: "02546e2b2b7443c1b44a97837a41836a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c95860436ba46bab2d945dacf32270e" is_pre_election: true
I20260806 10:58:38.834385 19368 raft_consensus.cc:2454] T 0893dfa0e92a40fc96b91236b03ae62c P 1da5ad6830b34ccaac7ea8194f8b850c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02546e2b2b7443c1b44a97837a41836a in term 0.
I20260806 10:58:38.834385 19093 raft_consensus.cc:2454] T 0893dfa0e92a40fc96b91236b03ae62c P 1c95860436ba46bab2d945dacf32270e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02546e2b2b7443c1b44a97837a41836a in term 0.
I20260806 10:58:38.835210 19162 leader_election.cc:302] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02546e2b2b7443c1b44a97837a41836a, 1da5ad6830b34ccaac7ea8194f8b850c; no voters:
I20260806 10:58:38.835896 19440 raft_consensus.cc:2781] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:58:38.836045 19440 raft_consensus.cc:482] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:58:38.836174 19440 raft_consensus.cc:3037] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:38.838469 19440 raft_consensus.cc:504] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da5ad6830b34ccaac7ea8194f8b850c" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 37739 } } peers { permanent_uuid: "02546e2b2b7443c1b44a97837a41836a" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42887 } } peers { permanent_uuid: "1c95860436ba46bab2d945dacf32270e" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 41219 } }
I20260806 10:58:38.839161 19440 leader_election.cc:288] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a [CANDIDATE]: Term 1 election: Requested vote from peers 1da5ad6830b34ccaac7ea8194f8b850c (127.30.218.3:37739), 1c95860436ba46bab2d945dacf32270e (127.30.218.1:41219)
I20260806 10:58:38.839708 19368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0893dfa0e92a40fc96b91236b03ae62c" candidate_uuid: "02546e2b2b7443c1b44a97837a41836a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da5ad6830b34ccaac7ea8194f8b850c"
I20260806 10:58:38.839774 19093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0893dfa0e92a40fc96b91236b03ae62c" candidate_uuid: "02546e2b2b7443c1b44a97837a41836a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c95860436ba46bab2d945dacf32270e"
I20260806 10:58:38.839928 19368 raft_consensus.cc:3037] T 0893dfa0e92a40fc96b91236b03ae62c P 1da5ad6830b34ccaac7ea8194f8b850c [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:38.840000 19093 raft_consensus.cc:3037] T 0893dfa0e92a40fc96b91236b03ae62c P 1c95860436ba46bab2d945dacf32270e [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:38.842358 19368 raft_consensus.cc:2454] T 0893dfa0e92a40fc96b91236b03ae62c P 1da5ad6830b34ccaac7ea8194f8b850c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02546e2b2b7443c1b44a97837a41836a in term 1.
I20260806 10:58:38.842543 19093 raft_consensus.cc:2454] T 0893dfa0e92a40fc96b91236b03ae62c P 1c95860436ba46bab2d945dacf32270e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02546e2b2b7443c1b44a97837a41836a in term 1.
I20260806 10:58:38.842885 19162 leader_election.cc:302] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02546e2b2b7443c1b44a97837a41836a, 1da5ad6830b34ccaac7ea8194f8b850c; no voters:
I20260806 10:58:38.843242 19440 raft_consensus.cc:2781] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:58:38.843698 19440 raft_consensus.cc:686] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a [term 1 LEADER]: Becoming Leader. State: Replica: 02546e2b2b7443c1b44a97837a41836a, State: Running, Role: LEADER
I20260806 10:58:38.844246 19440 consensus_queue.cc:237] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da5ad6830b34ccaac7ea8194f8b850c" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 37739 } } peers { permanent_uuid: "02546e2b2b7443c1b44a97837a41836a" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42887 } } peers { permanent_uuid: "1c95860436ba46bab2d945dacf32270e" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 41219 } }
I20260806 10:58:38.849565 18944 catalog_manager.cc:5495] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a reported cstate change: term changed from 0 to 1, leader changed from <none> to 02546e2b2b7443c1b44a97837a41836a (127.30.218.2). New cstate: current_term: 1 leader_uuid: "02546e2b2b7443c1b44a97837a41836a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da5ad6830b34ccaac7ea8194f8b850c" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 37739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02546e2b2b7443c1b44a97837a41836a" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 42887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c95860436ba46bab2d945dacf32270e" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 41219 } health_report { overall_health: UNKNOWN } } }
I20260806 10:58:38.936556 19433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:38.930658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42518 (local address 127.30.218.2:42887)
0806 10:58:38.930908 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:38.930914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:38.931099 (+ 185us) server_negotiation.cc:407] Connection header received
0806 10:58:38.931337 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:38.931342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:38.931422 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:38.931566 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:58:38.932044 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.932764 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.934304 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:38.934757 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:38.934886 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:38.935392 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:58:38.935913 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:58:38.936057 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:38.936163 (+ 106us) server_negotiation.cc:299] Negotiation successful
0806 10:58:38.936268 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 10:58:38.945114 19368 raft_consensus.cc:1261] T 0893dfa0e92a40fc96b91236b03ae62c P 1da5ad6830b34ccaac7ea8194f8b850c [term 1 FOLLOWER]: Refusing update from remote peer 02546e2b2b7443c1b44a97837a41836a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:58:38.945123 19093 raft_consensus.cc:1261] T 0893dfa0e92a40fc96b91236b03ae62c P 1c95860436ba46bab2d945dacf32270e [term 1 FOLLOWER]: Refusing update from remote peer 02546e2b2b7443c1b44a97837a41836a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:58:38.946420 19445 consensus_queue.cc:1035] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c95860436ba46bab2d945dacf32270e" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:58:38.946852 19440 consensus_queue.cc:1035] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da5ad6830b34ccaac7ea8194f8b850c" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 37739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:58:38.957417 19450 mvcc.cc:205] Tried to move back new op lower bound from 7315513011988307968 to 7315513011590270976. Current Snapshot: MvccSnapshot[applied={T|T < 7315513011988307968}]
I20260806 10:58:38.962356 19453 mvcc.cc:205] Tried to move back new op lower bound from 7315513011988307968 to 7315513011590270976. Current Snapshot: MvccSnapshot[applied={T|T < 7315513011988307968 or (T in {7315513011988307968})}]
I20260806 10:58:38.973589 18942 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:58:38.973924 18942 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:58:38.977869 18942 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75b00ee907c14c04ad589bda0a9d7a45: Sending DeleteTablet for 3 replicas of tablet 0893dfa0e92a40fc96b91236b03ae62c
I20260806 10:58:38.979354 19072 tablet_service.cc:1514] Processing DeleteTablet for tablet 0893dfa0e92a40fc96b91236b03ae62c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:58:38 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:34886
I20260806 10:58:38.979434 19209 tablet_service.cc:1514] Processing DeleteTablet for tablet 0893dfa0e92a40fc96b91236b03ae62c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:58:38 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:42516
I20260806 10:58:38.979411 19347 tablet_service.cc:1514] Processing DeleteTablet for tablet 0893dfa0e92a40fc96b91236b03ae62c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:58:38 UTC) from {username='kudu', principal='kudu/127.30.218.62@KRBTEST.COM'} at 127.0.0.1:32818
I20260806 10:58:38.985510 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 19007
I20260806 10:58:38.992288 19461 tablet_replica.cc:330] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a: stopping tablet replica
I20260806 10:58:38.992905 19461 raft_consensus.cc:2229] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:58:38.993791 19461 raft_consensus.cc:2258] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:58:38.998462 19461 ts_tablet_manager.cc:1905] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:58:39.000723 19462 tablet_replica.cc:330] T 0893dfa0e92a40fc96b91236b03ae62c P 1da5ad6830b34ccaac7ea8194f8b850c: stopping tablet replica
I20260806 10:58:39.001194 19462 raft_consensus.cc:2229] T 0893dfa0e92a40fc96b91236b03ae62c P 1da5ad6830b34ccaac7ea8194f8b850c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:58:39.001766 19462 raft_consensus.cc:2258] T 0893dfa0e92a40fc96b91236b03ae62c P 1da5ad6830b34ccaac7ea8194f8b850c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:58:39.004736 19460 tablet_replica.cc:330] T 0893dfa0e92a40fc96b91236b03ae62c P 1c95860436ba46bab2d945dacf32270e: stopping tablet replica
I20260806 10:58:39.006104 19462 ts_tablet_manager.cc:1905] T 0893dfa0e92a40fc96b91236b03ae62c P 1da5ad6830b34ccaac7ea8194f8b850c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:58:39.006647 19460 raft_consensus.cc:2229] T 0893dfa0e92a40fc96b91236b03ae62c P 1c95860436ba46bab2d945dacf32270e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:58:39.007521 19460 raft_consensus.cc:2258] T 0893dfa0e92a40fc96b91236b03ae62c P 1c95860436ba46bab2d945dacf32270e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:58:39.009341 19461 ts_tablet_manager.cc:1918] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:58:39.009554 19461 log.cc:1196] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/0893dfa0e92a40fc96b91236b03ae62c
I20260806 10:58:39.010121 19461 ts_tablet_manager.cc:1939] T 0893dfa0e92a40fc96b91236b03ae62c P 02546e2b2b7443c1b44a97837a41836a: Deleting consensus metadata
I20260806 10:58:39.012457 18929 catalog_manager.cc:4841] TS 02546e2b2b7443c1b44a97837a41836a (127.30.218.2:42887): tablet 0893dfa0e92a40fc96b91236b03ae62c (table test-table [id=dc1f237e3e97400e9eb2881738427378]) successfully deleted
I20260806 10:58:39.013634 19460 ts_tablet_manager.cc:1905] T 0893dfa0e92a40fc96b91236b03ae62c P 1c95860436ba46bab2d945dacf32270e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:58:39.013725 19462 ts_tablet_manager.cc:1918] T 0893dfa0e92a40fc96b91236b03ae62c P 1da5ad6830b34ccaac7ea8194f8b850c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:58:39.013906 19462 log.cc:1196] T 0893dfa0e92a40fc96b91236b03ae62c P 1da5ad6830b34ccaac7ea8194f8b850c: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/0893dfa0e92a40fc96b91236b03ae62c
I20260806 10:58:39.014456 19462 ts_tablet_manager.cc:1939] T 0893dfa0e92a40fc96b91236b03ae62c P 1da5ad6830b34ccaac7ea8194f8b850c: Deleting consensus metadata
I20260806 10:58:39.016435 18929 catalog_manager.cc:4841] TS 1da5ad6830b34ccaac7ea8194f8b850c (127.30.218.3:37739): tablet 0893dfa0e92a40fc96b91236b03ae62c (table test-table [id=dc1f237e3e97400e9eb2881738427378]) successfully deleted
Aug 06 10:58:39 dist-test-slave-49h3 krb5kdc[18898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.1@KRBTEST.COM
I20260806 10:58:39.019500 19460 ts_tablet_manager.cc:1918] T 0893dfa0e92a40fc96b91236b03ae62c P 1c95860436ba46bab2d945dacf32270e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:58:39.019675 19460 log.cc:1196] T 0893dfa0e92a40fc96b91236b03ae62c P 1c95860436ba46bab2d945dacf32270e: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786013589554914-31592-0/minicluster-data/ts-0/wal/wals/0893dfa0e92a40fc96b91236b03ae62c
I20260806 10:58:39.020102 19460 ts_tablet_manager.cc:1939] T 0893dfa0e92a40fc96b91236b03ae62c P 1c95860436ba46bab2d945dacf32270e: Deleting consensus metadata
I20260806 10:58:39.021981 18931 catalog_manager.cc:4841] TS 1c95860436ba46bab2d945dacf32270e (127.30.218.1:41219): tablet 0893dfa0e92a40fc96b91236b03ae62c (table test-table [id=dc1f237e3e97400e9eb2881738427378]) successfully deleted
I20260806 10:58:39.026556 19432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:38.987428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34888 (local address 127.30.218.1:41219)
0806 10:58:38.987625 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:38.987633 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:38.998376 (+ 10743us) server_negotiation.cc:407] Connection header received
0806 10:58:38.999005 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:38.999011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:38.999157 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:38.999238 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:39.001251 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:39.002420 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:39.004938 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:39.005551 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:39.005652 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:39.019274 (+ 13622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:39.019292 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:39.019312 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:39.019362 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:39.023506 (+ 4144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:39.024313 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:39.024324 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:39.024328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:39.024511 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:39.024865 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:39.024870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:39.024874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:39.025337 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:39.025675 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:39.025915 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 10:58:39.026168 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 10:58:39.027406 19133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:39.118219 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 19007
I20260806 10:58:39.136323 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 19145
Aug 06 10:58:39 dist-test-slave-49h3 krb5kdc[18898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.2@KRBTEST.COM
I20260806 10:58:39.154677 19433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:39.137086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42528 (local address 127.30.218.2:42887)
0806 10:58:39.137271 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:39.137276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:39.137372 (+ 96us) server_negotiation.cc:407] Connection header received
0806 10:58:39.137605 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:39.137610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:39.137707 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:39.137794 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:39.139507 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:39.140366 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:39.142419 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:39.142882 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:39.142996 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:39.148396 (+ 5400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:39.148412 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:39.148427 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:39.148468 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:39.151666 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:39.152460 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:39.152469 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:39.152471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:39.152624 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:39.152999 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:39.153003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:39.153005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:39.153420 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:39.153823 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:39.154087 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 10:58:39.154313 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 10:58:39.155365 19271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:39.274837 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 19145
I20260806 10:58:39.293015 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 19283
Aug 06 10:58:39 dist-test-slave-49h3 krb5kdc[18898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.3@KRBTEST.COM
I20260806 10:58:39.308523 19429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:39.293784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32830 (local address 127.30.218.3:37739)
0806 10:58:39.293948 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:39.293953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:39.293973 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:58:39.294130 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:39.294134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:39.294213 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:39.294315 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:39.295557 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:39.296277 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:39.297519 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:39.298034 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:39.298133 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:39.302889 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:39.302907 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:39.302925 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:39.302970 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:39.305732 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:39.306565 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:39.306574 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:39.306576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:39.306752 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:39.307090 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:39.307094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:39.307097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:39.307556 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:39.307787 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:39.308036 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 10:58:39.308243 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 10:58:39.309201 19408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:39.400472 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 19283
I20260806 10:58:39.417236 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 18911
I20260806 10:58:39.418476 18975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:39.530215 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 18911
2026-08-06T10:58:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4812 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T10:58:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:58:39Z Disabled control of system clock
I20260806 10:58:39.625110 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:38559
--webserver_interface=127.30.218.62
--webserver_port=0
--builtin_ntp_servers=127.30.218.20:34775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:38559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 10:58:39.984607 19471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:39.984947 19471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:39.985033 19471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:39.992805 19471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:58:39.992938 19471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:39.992976 19471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:39.993031 19471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:58:39.993075 19471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:58:40.002657 19471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:34775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:38559
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:38559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:40.003953 19471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:40.005791 19471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:40.017585 19479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:40.017661 19477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:40.017669 19476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:40.018567 19471 server_base.cc:1034] running on GCE node
I20260806 10:58:40.019400 19471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:40.020857 19471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:40.022066 19471 hybrid_clock.cc:648] HybridClock initialized: now 1786013920022021 us; error 34 us; skew 500 ppm
I20260806 10:58:40.022433 19471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:58:40.024780 19471 webserver.cc:460] Webserver started at http://127.30.218.62:46831/ using document root <none> and password file <none>
I20260806 10:58:40.025550 19471 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:40.025660 19471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:40.025995 19471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:40.029178 19471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "a60851f80f8241adb9810214aa3bc097"
format_stamp: "Formatted at 2026-08-06 10:58:40 on dist-test-slave-49h3"
server_key: "82a4ab9f78350881bebdcc5dd5a3899b"
server_key_iv: "90afaea1c1832b98283b8b7aaccf25d9"
server_key_version: "encryptionkey@0"
I20260806 10:58:40.030161 19471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "a60851f80f8241adb9810214aa3bc097"
format_stamp: "Formatted at 2026-08-06 10:58:40 on dist-test-slave-49h3"
server_key: "82a4ab9f78350881bebdcc5dd5a3899b"
server_key_iv: "90afaea1c1832b98283b8b7aaccf25d9"
server_key_version: "encryptionkey@0"
I20260806 10:58:40.036175 19471 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 10:58:40.040349 19485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:40.042011 19471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 10:58:40.042249 19471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "a60851f80f8241adb9810214aa3bc097"
format_stamp: "Formatted at 2026-08-06 10:58:40 on dist-test-slave-49h3"
server_key: "82a4ab9f78350881bebdcc5dd5a3899b"
server_key_iv: "90afaea1c1832b98283b8b7aaccf25d9"
server_key_version: "encryptionkey@0"
I20260806 10:58:40.042505 19471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:40.073099 19471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:40.074126 19471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:40.074435 19471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:40.093135 19471 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:38559
I20260806 10:58:40.093156 19537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:38559 every 8 connection(s)
I20260806 10:58:40.094563 19471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:58:40.098465 19538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:40.099404 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 19471
I20260806 10:58:40.099572 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:58:40.100039 31592 external_mini_cluster.cc:1428] Setting key a88e81b5521f22ab9497e677ff89a3b1
I20260806 10:58:40.113405 19538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a60851f80f8241adb9810214aa3bc097: Bootstrap starting.
I20260806 10:58:40.115803 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:40.102157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54456 (local address 127.30.218.62:38559)
0806 10:58:40.103324 (+ 1167us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:40.103340 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:40.103393 (+ 53us) server_negotiation.cc:407] Connection header received
0806 10:58:40.105202 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:40.105242 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:40.105925 (+ 683us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:40.106438 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:40.106969 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:40.108822 (+ 1853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:40.110332 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:40.111465 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:40.111648 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:40.112784 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:40.112795 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:40.112814 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:40.113587 (+ 773us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:40.113696 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:40.113725 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:40.113864 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:40.114095 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 10:58:40.114472 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":328,"threads_started":1}
I20260806 10:58:40.117393 19538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a60851f80f8241adb9810214aa3bc097: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:40.118573 19538 log.cc:824] T 00000000000000000000000000000000 P a60851f80f8241adb9810214aa3bc097: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:40.121614 19538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a60851f80f8241adb9810214aa3bc097: No bootstrap required, opened a new log
I20260806 10:58:40.129364 19538 raft_consensus.cc:348] T 00000000000000000000000000000000 P a60851f80f8241adb9810214aa3bc097 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60851f80f8241adb9810214aa3bc097" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 38559 } }
I20260806 10:58:40.129916 19538 raft_consensus.cc:374] T 00000000000000000000000000000000 P a60851f80f8241adb9810214aa3bc097 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:40.130033 19538 raft_consensus.cc:729] T 00000000000000000000000000000000 P a60851f80f8241adb9810214aa3bc097 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60851f80f8241adb9810214aa3bc097, State: Initialized, Role: FOLLOWER
I20260806 10:58:40.130712 19538 consensus_queue.cc:260] T 00000000000000000000000000000000 P a60851f80f8241adb9810214aa3bc097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60851f80f8241adb9810214aa3bc097" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 38559 } }
I20260806 10:58:40.130986 19538 raft_consensus.cc:388] T 00000000000000000000000000000000 P a60851f80f8241adb9810214aa3bc097 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:58:40.131135 19538 raft_consensus.cc:482] T 00000000000000000000000000000000 P a60851f80f8241adb9810214aa3bc097 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:58:40.131312 19538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a60851f80f8241adb9810214aa3bc097 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:40.133677 19538 raft_consensus.cc:504] T 00000000000000000000000000000000 P a60851f80f8241adb9810214aa3bc097 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60851f80f8241adb9810214aa3bc097" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 38559 } }
I20260806 10:58:40.134259 19538 leader_election.cc:302] T 00000000000000000000000000000000 P a60851f80f8241adb9810214aa3bc097 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a60851f80f8241adb9810214aa3bc097; no voters:
I20260806 10:58:40.134837 19538 leader_election.cc:288] T 00000000000000000000000000000000 P a60851f80f8241adb9810214aa3bc097 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:58:40.134997 19544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a60851f80f8241adb9810214aa3bc097 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:58:40.135788 19544 raft_consensus.cc:686] T 00000000000000000000000000000000 P a60851f80f8241adb9810214aa3bc097 [term 1 LEADER]: Becoming Leader. State: Replica: a60851f80f8241adb9810214aa3bc097, State: Running, Role: LEADER
I20260806 10:58:40.136359 19544 consensus_queue.cc:237] T 00000000000000000000000000000000 P a60851f80f8241adb9810214aa3bc097 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60851f80f8241adb9810214aa3bc097" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 38559 } }
I20260806 10:58:40.137042 19538 sys_catalog.cc:564] T 00000000000000000000000000000000 P a60851f80f8241adb9810214aa3bc097 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:58:40.139107 19547 sys_catalog.cc:455] T 00000000000000000000000000000000 P a60851f80f8241adb9810214aa3bc097 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a60851f80f8241adb9810214aa3bc097. Latest consensus state: current_term: 1 leader_uuid: "a60851f80f8241adb9810214aa3bc097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60851f80f8241adb9810214aa3bc097" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 38559 } } }
I20260806 10:58:40.139541 19547 sys_catalog.cc:458] T 00000000000000000000000000000000 P a60851f80f8241adb9810214aa3bc097 [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:40.139295 19546 sys_catalog.cc:455] T 00000000000000000000000000000000 P a60851f80f8241adb9810214aa3bc097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a60851f80f8241adb9810214aa3bc097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60851f80f8241adb9810214aa3bc097" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 38559 } } }
I20260806 10:58:40.139976 19546 sys_catalog.cc:458] T 00000000000000000000000000000000 P a60851f80f8241adb9810214aa3bc097 [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:40.143261 19550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:58:40.149891 19550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:58:40.160928 19550 catalog_manager.cc:1347] Generated new cluster ID: e8a75e9c09f449c683a1f3353cba6a5b
I20260806 10:58:40.161077 19550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:58:40.181860 19550 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:58:40.183991 19550 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:58:40.208251 19550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a60851f80f8241adb9810214aa3bc097: Generated new TSK 0
I20260806 10:58:40.209481 19550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:58:40.221189 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.1:0
--local_ip_for_outbound_sockets=127.30.218.1
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:38559
--builtin_ntp_servers=127.30.218.20:34775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 10:58:40.576295 19564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:40.576609 19564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:40.576738 19564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:40.584296 19564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:40.584434 19564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:40.584591 19564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.1
I20260806 10:58:40.594161 19564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:34775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.218.1
--webserver_port=0
--tserver_master_addrs=127.30.218.62:38559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.1
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:40.595517 19564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:40.597347 19564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:40.610496 19571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:40.610657 19569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:40.610884 19573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:40.612035 19564 server_base.cc:1034] running on GCE node
I20260806 10:58:40.612725 19564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:40.613783 19564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:40.615005 19564 hybrid_clock.cc:648] HybridClock initialized: now 1786013920614955 us; error 46 us; skew 500 ppm
I20260806 10:58:40.615345 19564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:58:40.617666 19564 webserver.cc:460] Webserver started at http://127.30.218.1:45447/ using document root <none> and password file <none>
I20260806 10:58:40.618363 19564 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:40.618459 19564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:40.618764 19564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:40.621732 19564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-0/data/instance:
uuid: "eb5ef90d8dc04825b62fa75e2ee4261d"
format_stamp: "Formatted at 2026-08-06 10:58:40 on dist-test-slave-49h3"
server_key: "5213521aa3a22007706f7a5065976502"
server_key_iv: "c36cefcca6a995c7f30566731157df48"
server_key_version: "encryptionkey@0"
I20260806 10:58:40.622565 19564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance:
uuid: "eb5ef90d8dc04825b62fa75e2ee4261d"
format_stamp: "Formatted at 2026-08-06 10:58:40 on dist-test-slave-49h3"
server_key: "5213521aa3a22007706f7a5065976502"
server_key_iv: "c36cefcca6a995c7f30566731157df48"
server_key_version: "encryptionkey@0"
I20260806 10:58:40.628067 19564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 10:58:40.631772 19579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:40.633190 19564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:58:40.633353 19564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-0/wal
uuid: "eb5ef90d8dc04825b62fa75e2ee4261d"
format_stamp: "Formatted at 2026-08-06 10:58:40 on dist-test-slave-49h3"
server_key: "5213521aa3a22007706f7a5065976502"
server_key_iv: "c36cefcca6a995c7f30566731157df48"
server_key_version: "encryptionkey@0"
I20260806 10:58:40.633630 19564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:40.656613 19564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:40.657518 19564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:40.657833 19564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:40.658828 19564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:58:40.660274 19564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:58:40.660373 19564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:40.660477 19564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:58:40.660545 19564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:40.690714 19564 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.1:42511
I20260806 10:58:40.690785 19695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.1:42511 every 8 connection(s)
I20260806 10:58:40.692011 19564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-0/data/info.pb
I20260806 10:58:40.698364 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 19564
I20260806 10:58:40.698536 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-0/wal/instance
I20260806 10:58:40.698977 31592 external_mini_cluster.cc:1428] Setting key 7839783089880a2d5a45507a4fbd4f28
I20260806 10:58:40.701751 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.2:0
--local_ip_for_outbound_sockets=127.30.218.2
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:38559
--builtin_ntp_servers=127.30.218.20:34775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 10:58:40.704545 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:40.694563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.1:57401 (local address 127.30.218.62:38559)
0806 10:58:40.695007 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:40.695011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:40.696055 (+ 1044us) server_negotiation.cc:407] Connection header received
0806 10:58:40.697934 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:40.697939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:40.698024 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:40.698122 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:40.699320 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:40.700081 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:40.701724 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:40.702192 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:40.702311 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:40.703697 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:40.703702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:40.703705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:40.703725 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:40.703766 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:40.703779 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:40.703857 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:40.704078 (+ 221us) server_negotiation.cc:299] Negotiation successful
0806 10:58:40.704184 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":224,"threads_started":1}
I20260806 10:58:40.705655 19700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:40.695030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:38559 (local address 127.30.218.1:57401)
0806 10:58:40.695901 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:40.695936 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:40.697655 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:40.698368 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:40.698377 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:40.698486 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:58:40.699136 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:40.699155 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:40.700249 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:40.700253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:40.701578 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:40.701591 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:40.702424 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:40.702431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:40.702681 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:40.703346 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:58:40.703349 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:58:40.703389 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:58:40.703478 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:58:40.703871 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:58:40.703884 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:58:40.703891 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 10:58:40.704055 (+ 164us) client_negotiation.cc:240] Negotiation successful
0806 10:58:40.704502 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":269,"threads_started":1}
I20260806 10:58:40.707422 19696 heartbeater.cc:347] Connected to a master server at 127.30.218.62:38559
I20260806 10:58:40.707839 19696 heartbeater.cc:464] Registering TS with master...
I20260806 10:58:40.708630 19696 heartbeater.cc:511] Master 127.30.218.62:38559 requested a full tablet report, sending...
I20260806 10:58:40.711100 19502 ts_manager.cc:194] Registered new tserver with Master: eb5ef90d8dc04825b62fa75e2ee4261d (127.30.218.1:42511)
I20260806 10:58:40.712973 19502 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.218.1:57401
W20260806 10:58:41.044994 19701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:41.045284 19701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:41.045395 19701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:41.052829 19701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:41.052925 19701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:41.053067 19701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.2
I20260806 10:58:41.062058 19701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:34775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.218.2
--webserver_port=0
--tserver_master_addrs=127.30.218.62:38559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.2
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:41.063274 19701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:41.065029 19701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:41.077863 19706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:41.077890 19709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:41.077875 19707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:41.079043 19701 server_base.cc:1034] running on GCE node
I20260806 10:58:41.079654 19701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:41.080638 19701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:41.081857 19701 hybrid_clock.cc:648] HybridClock initialized: now 1786013921081809 us; error 34 us; skew 500 ppm
I20260806 10:58:41.082207 19701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:58:41.084640 19701 webserver.cc:460] Webserver started at http://127.30.218.2:39981/ using document root <none> and password file <none>
I20260806 10:58:41.085382 19701 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:41.085511 19701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:41.085862 19701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:41.088941 19701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-1/data/instance:
uuid: "8330cb2306b043ac8752cf6d47b0c416"
format_stamp: "Formatted at 2026-08-06 10:58:41 on dist-test-slave-49h3"
server_key: "db00a9388dfee4c794825d02d9901ef3"
server_key_iv: "658ee368d1ed4d476929d98eaddffa96"
server_key_version: "encryptionkey@0"
I20260806 10:58:41.089830 19701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance:
uuid: "8330cb2306b043ac8752cf6d47b0c416"
format_stamp: "Formatted at 2026-08-06 10:58:41 on dist-test-slave-49h3"
server_key: "db00a9388dfee4c794825d02d9901ef3"
server_key_iv: "658ee368d1ed4d476929d98eaddffa96"
server_key_version: "encryptionkey@0"
I20260806 10:58:41.095588 19701 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:58:41.099848 19715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:41.101423 19701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 10:58:41.101655 19701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-1/wal
uuid: "8330cb2306b043ac8752cf6d47b0c416"
format_stamp: "Formatted at 2026-08-06 10:58:41 on dist-test-slave-49h3"
server_key: "db00a9388dfee4c794825d02d9901ef3"
server_key_iv: "658ee368d1ed4d476929d98eaddffa96"
server_key_version: "encryptionkey@0"
I20260806 10:58:41.101893 19701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:41.117075 19701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:41.118063 19701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:41.118379 19701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:41.119459 19701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:58:41.121091 19701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:58:41.121199 19701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:41.121322 19701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:58:41.121393 19701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:41.153846 19701 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.2:39015
I20260806 10:58:41.153956 19833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.2:39015 every 8 connection(s)
I20260806 10:58:41.155236 19701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-1/data/info.pb
I20260806 10:58:41.156350 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 19701
I20260806 10:58:41.156526 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-1/wal/instance
I20260806 10:58:41.156945 31592 external_mini_cluster.cc:1428] Setting key f12a8312a7d4ceedbea87728f3ba34d9
I20260806 10:58:41.159948 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.218.3:0
--local_ip_for_outbound_sockets=127.30.218.3
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:38559
--builtin_ntp_servers=127.30.218.20:34775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 10:58:41.167667 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:41.157950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:53147 (local address 127.30.218.62:38559)
0806 10:58:41.158165 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:41.158171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:41.159595 (+ 1424us) server_negotiation.cc:407] Connection header received
0806 10:58:41.161392 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:41.161397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:41.161507 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:41.161607 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:41.162681 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.163402 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.164935 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.165341 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.165476 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:41.166832 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:41.166838 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:41.166841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:41.166861 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:41.166893 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:41.166905 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:41.167007 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:41.167233 (+ 226us) server_negotiation.cc:299] Negotiation successful
0806 10:58:41.167337 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 10:58:41.168690 19836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:41.158462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:38559 (local address 127.30.218.2:53147)
0806 10:58:41.159438 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:41.159482 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:41.161136 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:41.161836 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:41.161844 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:41.161932 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:58:41.162515 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:41.162532 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.163547 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.163550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:41.164792 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:41.164805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.165587 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.165594 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:41.165817 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:41.166519 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:58:41.166522 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:58:41.166564 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:58:41.166643 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:58:41.167017 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:58:41.167028 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:58:41.167036 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 10:58:41.167209 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 10:58:41.167613 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":340,"threads_started":1}
I20260806 10:58:41.170073 19834 heartbeater.cc:347] Connected to a master server at 127.30.218.62:38559
I20260806 10:58:41.170365 19834 heartbeater.cc:464] Registering TS with master...
I20260806 10:58:41.171121 19834 heartbeater.cc:511] Master 127.30.218.62:38559 requested a full tablet report, sending...
I20260806 10:58:41.172816 19502 ts_manager.cc:194] Registered new tserver with Master: 8330cb2306b043ac8752cf6d47b0c416 (127.30.218.2:39015)
I20260806 10:58:41.173683 19502 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.218.2:53147
W20260806 10:58:41.502776 19837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:41.503094 19837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:41.503187 19837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:41.510964 19837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:41.511094 19837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:41.511243 19837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.218.3
I20260806 10:58:41.520805 19837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:34775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.218.3
--webserver_port=0
--tserver_master_addrs=127.30.218.62:38559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.218.3
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:41.522233 19837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:41.524132 19837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:41.537768 19843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:41.537876 19842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:41.537817 19845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:41.539264 19837 server_base.cc:1034] running on GCE node
I20260806 10:58:41.539901 19837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:41.540861 19837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:41.542059 19837 hybrid_clock.cc:648] HybridClock initialized: now 1786013921542003 us; error 45 us; skew 500 ppm
I20260806 10:58:41.542378 19837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 10:58:41.544584 19837 webserver.cc:460] Webserver started at http://127.30.218.3:45987/ using document root <none> and password file <none>
I20260806 10:58:41.545257 19837 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:41.545346 19837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:41.545660 19837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:41.548660 19837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-2/data/instance:
uuid: "204975809a1e45aa8800c1148c4730dd"
format_stamp: "Formatted at 2026-08-06 10:58:41 on dist-test-slave-49h3"
server_key: "d4109dc54627af13d2c4e8543095c56e"
server_key_iv: "c84495af84b573a5dc606a0e720849d7"
server_key_version: "encryptionkey@0"
I20260806 10:58:41.549403 19837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance:
uuid: "204975809a1e45aa8800c1148c4730dd"
format_stamp: "Formatted at 2026-08-06 10:58:41 on dist-test-slave-49h3"
server_key: "d4109dc54627af13d2c4e8543095c56e"
server_key_iv: "c84495af84b573a5dc606a0e720849d7"
server_key_version: "encryptionkey@0"
I20260806 10:58:41.554420 19837 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 10:58:41.557780 19851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:41.559072 19837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 10:58:41.559213 19837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-2/wal
uuid: "204975809a1e45aa8800c1148c4730dd"
format_stamp: "Formatted at 2026-08-06 10:58:41 on dist-test-slave-49h3"
server_key: "d4109dc54627af13d2c4e8543095c56e"
server_key_iv: "c84495af84b573a5dc606a0e720849d7"
server_key_version: "encryptionkey@0"
I20260806 10:58:41.559396 19837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:41.582650 19837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:41.583550 19837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:41.583840 19837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:41.584851 19837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 10:58:41.586474 19837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 10:58:41.586568 19837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:41.586665 19837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 10:58:41.586719 19837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:41.618454 19837 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.3:41883
I20260806 10:58:41.618507 19967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.3:41883 every 8 connection(s)
I20260806 10:58:41.619745 19837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-2/data/info.pb
I20260806 10:58:41.623526 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 19837
I20260806 10:58:41.623718 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-2/wal/instance
I20260806 10:58:41.624218 31592 external_mini_cluster.cc:1428] Setting key fe3ab7ef6c0d8539f8eec27e1abfef44
I20260806 10:58:41.632694 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:41.622233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.3:57569 (local address 127.30.218.62:38559)
0806 10:58:41.622408 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:41.622412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:41.623830 (+ 1418us) server_negotiation.cc:407] Connection header received
0806 10:58:41.626058 (+ 2228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:41.626064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:41.626161 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:41.626252 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:41.627612 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.628362 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.630040 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.630479 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.630606 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:41.631984 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:41.631989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:41.631992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:41.632013 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:41.632052 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:41.632065 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:41.632141 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:41.632327 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 10:58:41.632422 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 10:58:41.633976 19970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:41.622736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:38559 (local address 127.30.218.3:57569)
0806 10:58:41.623648 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:41.623707 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:41.625744 (+ 2037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:41.626533 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:41.626546 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:41.626670 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:58:41.627456 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:41.627478 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.628486 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.628490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:41.629874 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:41.629889 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.630698 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.630704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:41.630933 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:41.631636 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 10:58:41.631639 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:58:41.631682 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 10:58:41.631784 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:58:41.632141 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:58:41.632148 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:58:41.632153 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 10:58:41.632313 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 10:58:41.632665 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":278,"threads_started":1}
I20260806 10:58:41.635584 19968 heartbeater.cc:347] Connected to a master server at 127.30.218.62:38559
I20260806 10:58:41.635897 19968 heartbeater.cc:464] Registering TS with master...
I20260806 10:58:41.636715 19968 heartbeater.cc:511] Master 127.30.218.62:38559 requested a full tablet report, sending...
I20260806 10:58:41.638423 19502 ts_manager.cc:194] Registered new tserver with Master: 204975809a1e45aa8800c1148c4730dd (127.30.218.3:41883)
I20260806 10:58:41.639415 19502 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.218.3:57569
I20260806 10:58:41.640026 31592 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 10:58:41.650974 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:41.645110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54472 (local address 127.30.218.62:38559)
0806 10:58:41.645513 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:41.645521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:41.645556 (+ 35us) server_negotiation.cc:407] Connection header received
0806 10:58:41.645625 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:41.645630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:41.645725 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:41.645826 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:41.646438 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.647211 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.648602 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.649039 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.649158 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:41.650307 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:41.650313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:41.650316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:41.650340 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:41.650375 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:41.650390 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:41.650471 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:41.650629 (+ 158us) server_negotiation.cc:299] Negotiation successful
0806 10:58:41.650726 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 10:58:41.655284 19502 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 10:58:41.662395 19502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 10:58:41.687949 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:41.678081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53458 (local address 127.30.218.1:42511)
0806 10:58:41.679601 (+ 1520us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:41.679612 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:41.680162 (+ 550us) server_negotiation.cc:407] Connection header received
0806 10:58:41.680337 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:41.680346 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:41.680613 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:41.680802 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:41.681434 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.682844 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.685115 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.686680 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.686821 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:41.687071 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:41.687416 (+ 345us) server_negotiation.cc:299] Negotiation successful
0806 10:58:41.687612 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1332,"thread_start_us":1184,"threads_started":1}
I20260806 10:58:41.687939 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:41.677819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:42511 (local address 127.0.0.1:53458)
0806 10:58:41.680027 (+ 2208us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:41.680045 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:41.680171 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:41.680827 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:41.680834 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:41.680865 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:41.681191 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:41.681215 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.682973 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.682978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:41.684850 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:41.684866 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.686925 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.686931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:41.687159 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:41.687220 (+ 61us) client_negotiation.cc:769] Sending connection context
0806 10:58:41.687356 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 10:58:41.687577 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2092,"thread_start_us":1958,"threads_started":1}
I20260806 10:58:41.688983 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:41.674948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.2:39015 (local address 127.0.0.1:38510)
0806 10:58:41.678929 (+ 3981us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:41.678954 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:41.679070 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:41.681263 (+ 2193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:41.681268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:41.681292 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:41.681652 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:41.681665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.683932 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.683937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:41.686116 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:41.686132 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.688008 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.688015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:41.688527 (+ 512us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:41.688554 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 10:58:41.688618 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 10:58:41.688714 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2679,"mutex_wait_us":294,"thread_start_us":1493,"threads_started":1}
I20260806 10:58:41.690346 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:41.675442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38510 (local address 127.30.218.2:39015)
0806 10:58:41.680695 (+ 5253us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:41.680704 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:41.680734 (+ 30us) server_negotiation.cc:407] Connection header received
0806 10:58:41.680842 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:41.680850 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:41.681062 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:41.681267 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:41.682149 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.683775 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.686375 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.687772 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.688132 (+ 360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:41.689643 (+ 1511us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:41.689726 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 10:58:41.689902 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5086,"thread_start_us":4941,"threads_started":1}
I20260806 10:58:41.692111 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:41.674887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:41883 (local address 127.0.0.1:52716)
0806 10:58:41.676916 (+ 2029us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:41.676977 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:41.677168 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:41.684900 (+ 7732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:41.684905 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:41.684936 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:41.685359 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:41.685373 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.687200 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.687206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:41.689245 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:41.689262 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.691342 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.691349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:41.691529 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:41.691562 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 10:58:41.691628 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 10:58:41.691767 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1729,"mutex_wait_us":1,"thread_start_us":2579,"threads_started":1}
I20260806 10:58:41.692451 19629 tablet_service.cc:1467] Processing CreateTablet for tablet 3e9c6aa3120b48168ddc7249a78d8153 (DEFAULT_TABLE table=test-table [id=13c2b7f7bd6c480fbf868fbb3897d4bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:58:41.693115 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:41.675096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52716 (local address 127.30.218.3:41883)
0806 10:58:41.684395 (+ 9299us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:41.684402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:41.684431 (+ 29us) server_negotiation.cc:407] Connection header received
0806 10:58:41.684506 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:41.684513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:41.684707 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:41.684878 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:41.685572 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.687000 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.689485 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.691131 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.692352 (+ 1221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:41.692626 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:41.692699 (+ 73us) server_negotiation.cc:299] Negotiation successful
0806 10:58:41.692831 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9148,"thread_start_us":8992,"threads_started":1}
I20260806 10:58:41.694751 19629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e9c6aa3120b48168ddc7249a78d8153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:41.696930 19767 tablet_service.cc:1467] Processing CreateTablet for tablet 3e9c6aa3120b48168ddc7249a78d8153 (DEFAULT_TABLE table=test-table [id=13c2b7f7bd6c480fbf868fbb3897d4bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:58:41.698498 19767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e9c6aa3120b48168ddc7249a78d8153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:41.696935 19901 tablet_service.cc:1467] Processing CreateTablet for tablet 3e9c6aa3120b48168ddc7249a78d8153 (DEFAULT_TABLE table=test-table [id=13c2b7f7bd6c480fbf868fbb3897d4bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 10:58:41.700486 19901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e9c6aa3120b48168ddc7249a78d8153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:41.710930 19986 tablet_bootstrap.cc:492] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416: Bootstrap starting.
I20260806 10:58:41.713301 19987 tablet_bootstrap.cc:492] T 3e9c6aa3120b48168ddc7249a78d8153 P eb5ef90d8dc04825b62fa75e2ee4261d: Bootstrap starting.
I20260806 10:58:41.716214 19696 heartbeater.cc:503] Master 127.30.218.62:38559 was elected leader, sending a full tablet report...
I20260806 10:58:41.716261 19986 tablet_bootstrap.cc:654] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:41.718021 19986 log.cc:824] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:41.720016 19987 tablet_bootstrap.cc:654] T 3e9c6aa3120b48168ddc7249a78d8153 P eb5ef90d8dc04825b62fa75e2ee4261d: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:41.721690 19987 log.cc:824] T 3e9c6aa3120b48168ddc7249a78d8153 P eb5ef90d8dc04825b62fa75e2ee4261d: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:41.723858 19988 tablet_bootstrap.cc:492] T 3e9c6aa3120b48168ddc7249a78d8153 P 204975809a1e45aa8800c1148c4730dd: Bootstrap starting.
I20260806 10:58:41.724924 19986 tablet_bootstrap.cc:492] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416: No bootstrap required, opened a new log
I20260806 10:58:41.725266 19986 ts_tablet_manager.cc:1397] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20260806 10:58:41.725296 19987 tablet_bootstrap.cc:492] T 3e9c6aa3120b48168ddc7249a78d8153 P eb5ef90d8dc04825b62fa75e2ee4261d: No bootstrap required, opened a new log
I20260806 10:58:41.725595 19987 ts_tablet_manager.cc:1397] T 3e9c6aa3120b48168ddc7249a78d8153 P eb5ef90d8dc04825b62fa75e2ee4261d: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260806 10:58:41.727532 19988 tablet_bootstrap.cc:654] T 3e9c6aa3120b48168ddc7249a78d8153 P 204975809a1e45aa8800c1148c4730dd: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:41.728775 19988 log.cc:824] T 3e9c6aa3120b48168ddc7249a78d8153 P 204975809a1e45aa8800c1148c4730dd: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:41.731887 19988 tablet_bootstrap.cc:492] T 3e9c6aa3120b48168ddc7249a78d8153 P 204975809a1e45aa8800c1148c4730dd: No bootstrap required, opened a new log
I20260806 10:58:41.732232 19988 ts_tablet_manager.cc:1397] T 3e9c6aa3120b48168ddc7249a78d8153 P 204975809a1e45aa8800c1148c4730dd: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260806 10:58:41.736305 19986 raft_consensus.cc:348] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204975809a1e45aa8800c1148c4730dd" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41883 } } peers { permanent_uuid: "8330cb2306b043ac8752cf6d47b0c416" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39015 } } peers { permanent_uuid: "eb5ef90d8dc04825b62fa75e2ee4261d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 42511 } }
I20260806 10:58:41.736306 19987 raft_consensus.cc:348] T 3e9c6aa3120b48168ddc7249a78d8153 P eb5ef90d8dc04825b62fa75e2ee4261d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204975809a1e45aa8800c1148c4730dd" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41883 } } peers { permanent_uuid: "8330cb2306b043ac8752cf6d47b0c416" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39015 } } peers { permanent_uuid: "eb5ef90d8dc04825b62fa75e2ee4261d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 42511 } }
I20260806 10:58:41.736819 19986 raft_consensus.cc:374] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:41.736908 19987 raft_consensus.cc:374] T 3e9c6aa3120b48168ddc7249a78d8153 P eb5ef90d8dc04825b62fa75e2ee4261d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:41.736930 19986 raft_consensus.cc:729] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8330cb2306b043ac8752cf6d47b0c416, State: Initialized, Role: FOLLOWER
I20260806 10:58:41.737007 19987 raft_consensus.cc:729] T 3e9c6aa3120b48168ddc7249a78d8153 P eb5ef90d8dc04825b62fa75e2ee4261d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb5ef90d8dc04825b62fa75e2ee4261d, State: Initialized, Role: FOLLOWER
I20260806 10:58:41.737709 19986 consensus_queue.cc:260] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204975809a1e45aa8800c1148c4730dd" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41883 } } peers { permanent_uuid: "8330cb2306b043ac8752cf6d47b0c416" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39015 } } peers { permanent_uuid: "eb5ef90d8dc04825b62fa75e2ee4261d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 42511 } }
I20260806 10:58:41.737700 19987 consensus_queue.cc:260] T 3e9c6aa3120b48168ddc7249a78d8153 P eb5ef90d8dc04825b62fa75e2ee4261d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204975809a1e45aa8800c1148c4730dd" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41883 } } peers { permanent_uuid: "8330cb2306b043ac8752cf6d47b0c416" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39015 } } peers { permanent_uuid: "eb5ef90d8dc04825b62fa75e2ee4261d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 42511 } }
I20260806 10:58:41.740556 19988 raft_consensus.cc:348] T 3e9c6aa3120b48168ddc7249a78d8153 P 204975809a1e45aa8800c1148c4730dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204975809a1e45aa8800c1148c4730dd" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41883 } } peers { permanent_uuid: "8330cb2306b043ac8752cf6d47b0c416" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39015 } } peers { permanent_uuid: "eb5ef90d8dc04825b62fa75e2ee4261d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 42511 } }
I20260806 10:58:41.741070 19988 raft_consensus.cc:374] T 3e9c6aa3120b48168ddc7249a78d8153 P 204975809a1e45aa8800c1148c4730dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:41.741178 19988 raft_consensus.cc:729] T 3e9c6aa3120b48168ddc7249a78d8153 P 204975809a1e45aa8800c1148c4730dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 204975809a1e45aa8800c1148c4730dd, State: Initialized, Role: FOLLOWER
I20260806 10:58:41.741827 19988 consensus_queue.cc:260] T 3e9c6aa3120b48168ddc7249a78d8153 P 204975809a1e45aa8800c1148c4730dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204975809a1e45aa8800c1148c4730dd" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41883 } } peers { permanent_uuid: "8330cb2306b043ac8752cf6d47b0c416" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39015 } } peers { permanent_uuid: "eb5ef90d8dc04825b62fa75e2ee4261d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 42511 } }
I20260806 10:58:41.745272 19834 heartbeater.cc:503] Master 127.30.218.62:38559 was elected leader, sending a full tablet report...
I20260806 10:58:41.746709 19986 ts_tablet_manager.cc:1428] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416: Time spent starting tablet: real 0.021s user 0.017s sys 0.005s
I20260806 10:58:41.748545 19987 ts_tablet_manager.cc:1428] T 3e9c6aa3120b48168ddc7249a78d8153 P eb5ef90d8dc04825b62fa75e2ee4261d: Time spent starting tablet: real 0.023s user 0.010s sys 0.012s
I20260806 10:58:41.749212 19968 heartbeater.cc:503] Master 127.30.218.62:38559 was elected leader, sending a full tablet report...
I20260806 10:58:41.750020 19988 ts_tablet_manager.cc:1428] T 3e9c6aa3120b48168ddc7249a78d8153 P 204975809a1e45aa8800c1148c4730dd: Time spent starting tablet: real 0.018s user 0.019s sys 0.000s
I20260806 10:58:41.764541 19993 raft_consensus.cc:482] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 10:58:41.764917 19993 raft_consensus.cc:504] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204975809a1e45aa8800c1148c4730dd" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41883 } } peers { permanent_uuid: "8330cb2306b043ac8752cf6d47b0c416" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39015 } } peers { permanent_uuid: "eb5ef90d8dc04825b62fa75e2ee4261d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 42511 } }
I20260806 10:58:41.766826 19993 leader_election.cc:288] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 204975809a1e45aa8800c1148c4730dd (127.30.218.3:41883), eb5ef90d8dc04825b62fa75e2ee4261d (127.30.218.1:42511)
I20260806 10:58:41.773531 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:41.767240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:57033 (local address 127.30.218.3:41883)
0806 10:58:41.767429 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:41.767435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:41.768065 (+ 630us) server_negotiation.cc:407] Connection header received
0806 10:58:41.768309 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:41.768315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:41.768397 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:41.768504 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:41.769168 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.769963 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.771794 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.772589 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.772764 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:41.772840 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:41.773115 (+ 275us) server_negotiation.cc:299] Negotiation successful
0806 10:58:41.773256 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 10:58:41.773630 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:41.767233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.3:41883 (local address 127.30.218.2:57033)
0806 10:58:41.767952 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:41.767972 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:41.768133 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:41.768576 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:41.768581 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:41.768611 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:41.769003 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:41.769015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.770098 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.770102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:41.771600 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:41.771613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.772793 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.772799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:41.772950 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:41.772977 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 10:58:41.773061 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 10:58:41.773207 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":1,"thread_start_us":370,"threads_started":1}
I20260806 10:58:41.774264 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:41.768144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:42511 (local address 127.30.218.2:35125)
0806 10:58:41.769039 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:41.769053 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 10:58:41.769174 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:58:41.769533 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:58:41.769537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:58:41.769562 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 10:58:41.769877 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:41.769887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.770876 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.770879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:41.772315 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:58:41.772328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.773535 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.773542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:58:41.773743 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:41.773779 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 10:58:41.773864 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 10:58:41.773991 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":25,"thread_start_us":561,"threads_started":1}
I20260806 10:58:41.774338 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:41.768399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.218.2:35125 (local address 127.30.218.1:42511)
0806 10:58:41.768646 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:41.768653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:41.769145 (+ 492us) server_negotiation.cc:407] Connection header received
0806 10:58:41.769313 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:41.769317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:41.769393 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:41.769513 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 10:58:41.770041 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.770735 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.772500 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.773302 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.773474 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:41.773545 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:41.773913 (+ 368us) server_negotiation.cc:299] Negotiation successful
0806 10:58:41.774059 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 10:58:41.774883 19922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e9c6aa3120b48168ddc7249a78d8153" candidate_uuid: "8330cb2306b043ac8752cf6d47b0c416" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "204975809a1e45aa8800c1148c4730dd" is_pre_election: true
I20260806 10:58:41.775406 19650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e9c6aa3120b48168ddc7249a78d8153" candidate_uuid: "8330cb2306b043ac8752cf6d47b0c416" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb5ef90d8dc04825b62fa75e2ee4261d" is_pre_election: true
I20260806 10:58:41.775591 19922 raft_consensus.cc:2454] T 3e9c6aa3120b48168ddc7249a78d8153 P 204975809a1e45aa8800c1148c4730dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8330cb2306b043ac8752cf6d47b0c416 in term 0.
I20260806 10:58:41.775964 19650 raft_consensus.cc:2454] T 3e9c6aa3120b48168ddc7249a78d8153 P eb5ef90d8dc04825b62fa75e2ee4261d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8330cb2306b043ac8752cf6d47b0c416 in term 0.
I20260806 10:58:41.776440 19719 leader_election.cc:302] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 204975809a1e45aa8800c1148c4730dd, 8330cb2306b043ac8752cf6d47b0c416; no voters:
I20260806 10:58:41.776845 19993 raft_consensus.cc:2781] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 10:58:41.777014 19993 raft_consensus.cc:482] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 10:58:41.777129 19993 raft_consensus.cc:3037] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:41.779455 19993 raft_consensus.cc:504] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204975809a1e45aa8800c1148c4730dd" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41883 } } peers { permanent_uuid: "8330cb2306b043ac8752cf6d47b0c416" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39015 } } peers { permanent_uuid: "eb5ef90d8dc04825b62fa75e2ee4261d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 42511 } }
I20260806 10:58:41.780062 19993 leader_election.cc:288] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416 [CANDIDATE]: Term 1 election: Requested vote from peers 204975809a1e45aa8800c1148c4730dd (127.30.218.3:41883), eb5ef90d8dc04825b62fa75e2ee4261d (127.30.218.1:42511)
I20260806 10:58:41.780635 19922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e9c6aa3120b48168ddc7249a78d8153" candidate_uuid: "8330cb2306b043ac8752cf6d47b0c416" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "204975809a1e45aa8800c1148c4730dd"
I20260806 10:58:41.780673 19650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e9c6aa3120b48168ddc7249a78d8153" candidate_uuid: "8330cb2306b043ac8752cf6d47b0c416" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb5ef90d8dc04825b62fa75e2ee4261d"
I20260806 10:58:41.780843 19922 raft_consensus.cc:3037] T 3e9c6aa3120b48168ddc7249a78d8153 P 204975809a1e45aa8800c1148c4730dd [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:41.780877 19650 raft_consensus.cc:3037] T 3e9c6aa3120b48168ddc7249a78d8153 P eb5ef90d8dc04825b62fa75e2ee4261d [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:41.783191 19922 raft_consensus.cc:2454] T 3e9c6aa3120b48168ddc7249a78d8153 P 204975809a1e45aa8800c1148c4730dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8330cb2306b043ac8752cf6d47b0c416 in term 1.
I20260806 10:58:41.783198 19650 raft_consensus.cc:2454] T 3e9c6aa3120b48168ddc7249a78d8153 P eb5ef90d8dc04825b62fa75e2ee4261d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8330cb2306b043ac8752cf6d47b0c416 in term 1.
I20260806 10:58:41.783627 19719 leader_election.cc:302] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 204975809a1e45aa8800c1148c4730dd, 8330cb2306b043ac8752cf6d47b0c416; no voters:
I20260806 10:58:41.783921 19993 raft_consensus.cc:2781] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:58:41.784376 19993 raft_consensus.cc:686] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416 [term 1 LEADER]: Becoming Leader. State: Replica: 8330cb2306b043ac8752cf6d47b0c416, State: Running, Role: LEADER
I20260806 10:58:41.784868 19993 consensus_queue.cc:237] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204975809a1e45aa8800c1148c4730dd" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41883 } } peers { permanent_uuid: "8330cb2306b043ac8752cf6d47b0c416" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39015 } } peers { permanent_uuid: "eb5ef90d8dc04825b62fa75e2ee4261d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 42511 } }
I20260806 10:58:41.790014 19500 catalog_manager.cc:5495] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8330cb2306b043ac8752cf6d47b0c416 (127.30.218.2). New cstate: current_term: 1 leader_uuid: "8330cb2306b043ac8752cf6d47b0c416" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204975809a1e45aa8800c1148c4730dd" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8330cb2306b043ac8752cf6d47b0c416" member_type: VOTER last_known_addr { host: "127.30.218.2" port: 39015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5ef90d8dc04825b62fa75e2ee4261d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 42511 } health_report { overall_health: UNKNOWN } } }
I20260806 10:58:41.811620 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:41.806327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38522 (local address 127.30.218.2:39015)
0806 10:58:41.806511 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:41.806518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:41.806608 (+ 90us) server_negotiation.cc:407] Connection header received
0806 10:58:41.806843 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:41.806847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:41.806936 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:41.807065 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:58:41.807439 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.808134 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.809500 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.809919 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.810036 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:41.810527 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:58:41.811025 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:58:41.811166 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:41.811277 (+ 111us) server_negotiation.cc:299] Negotiation successful
0806 10:58:41.811360 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 10:58:41.820029 19922 raft_consensus.cc:1261] T 3e9c6aa3120b48168ddc7249a78d8153 P 204975809a1e45aa8800c1148c4730dd [term 1 FOLLOWER]: Refusing update from remote peer 8330cb2306b043ac8752cf6d47b0c416: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:58:41.820034 19650 raft_consensus.cc:1261] T 3e9c6aa3120b48168ddc7249a78d8153 P eb5ef90d8dc04825b62fa75e2ee4261d [term 1 FOLLOWER]: Refusing update from remote peer 8330cb2306b043ac8752cf6d47b0c416: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:58:41.821086 19997 consensus_queue.cc:1035] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5ef90d8dc04825b62fa75e2ee4261d" member_type: VOTER last_known_addr { host: "127.30.218.1" port: 42511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:58:41.821439 19993 consensus_queue.cc:1035] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416 [LEADER]: Connected to new peer: Peer: permanent_uuid: "204975809a1e45aa8800c1148c4730dd" member_type: VOTER last_known_addr { host: "127.30.218.3" port: 41883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:58:41.847746 19500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:58:41.848031 19500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 10:58:41.851922 19500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a60851f80f8241adb9810214aa3bc097: Sending DeleteTablet for 3 replicas of tablet 3e9c6aa3120b48168ddc7249a78d8153
I20260806 10:58:41.853159 19767 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e9c6aa3120b48168ddc7249a78d8153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:58:41 UTC) from {username='slave'} at 127.0.0.1:38510
I20260806 10:58:41.853229 19901 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e9c6aa3120b48168ddc7249a78d8153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:58:41 UTC) from {username='slave'} at 127.0.0.1:52716
I20260806 10:58:41.854324 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 19564
I20260806 10:58:41.854290 19629 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e9c6aa3120b48168ddc7249a78d8153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 10:58:41 UTC) from {username='slave'} at 127.0.0.1:53458
I20260806 10:58:41.876368 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:41.855394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53460 (local address 127.30.218.1:42511)
0806 10:58:41.855577 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:41.855584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:41.866541 (+ 10957us) server_negotiation.cc:407] Connection header received
0806 10:58:41.867133 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:41.867141 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:41.867272 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:41.867353 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:41.868110 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.869372 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.871423 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:41.872120 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:41.872224 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:41.874390 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:41.874404 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:41.874426 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:41.875304 (+ 878us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:41.875382 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:41.875412 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:41.875513 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:41.875784 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 10:58:41.875930 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 10:58:41.877732 19691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:41.882791 20012 tablet_replica.cc:330] T 3e9c6aa3120b48168ddc7249a78d8153 P 204975809a1e45aa8800c1148c4730dd: stopping tablet replica
I20260806 10:58:41.883520 20012 raft_consensus.cc:2229] T 3e9c6aa3120b48168ddc7249a78d8153 P 204975809a1e45aa8800c1148c4730dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 10:58:41.884269 20012 raft_consensus.cc:2258] T 3e9c6aa3120b48168ddc7249a78d8153 P 204975809a1e45aa8800c1148c4730dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:58:41.885085 20011 tablet_replica.cc:330] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416: stopping tablet replica
I20260806 10:58:41.885684 20011 raft_consensus.cc:2229] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416 [term 1 LEADER]: Raft consensus shutting down.
I20260806 10:58:41.886649 20011 raft_consensus.cc:2258] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 10:58:41.889429 20012 ts_tablet_manager.cc:1905] T 3e9c6aa3120b48168ddc7249a78d8153 P 204975809a1e45aa8800c1148c4730dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:58:41.893093 20011 ts_tablet_manager.cc:1905] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 10:58:41.897990 20012 ts_tablet_manager.cc:1918] T 3e9c6aa3120b48168ddc7249a78d8153 P 204975809a1e45aa8800c1148c4730dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:58:41.898244 20012 log.cc:1196] T 3e9c6aa3120b48168ddc7249a78d8153 P 204975809a1e45aa8800c1148c4730dd: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-2/wal/wals/3e9c6aa3120b48168ddc7249a78d8153
I20260806 10:58:41.898814 20012 ts_tablet_manager.cc:1939] T 3e9c6aa3120b48168ddc7249a78d8153 P 204975809a1e45aa8800c1148c4730dd: Deleting consensus metadata
I20260806 10:58:41.901213 19488 catalog_manager.cc:4841] TS 204975809a1e45aa8800c1148c4730dd (127.30.218.3:41883): tablet 3e9c6aa3120b48168ddc7249a78d8153 (table test-table [id=13c2b7f7bd6c480fbf868fbb3897d4bb]) successfully deleted
I20260806 10:58:41.901880 20011 ts_tablet_manager.cc:1918] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 10:58:41.902149 20011 log.cc:1196] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416: Deleting WAL directory at /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786013589554914-31592-0/minicluster-data/ts-1/wal/wals/3e9c6aa3120b48168ddc7249a78d8153
I20260806 10:58:41.902817 20011 ts_tablet_manager.cc:1939] T 3e9c6aa3120b48168ddc7249a78d8153 P 8330cb2306b043ac8752cf6d47b0c416: Deleting consensus metadata
I20260806 10:58:41.904860 19486 catalog_manager.cc:4841] TS 8330cb2306b043ac8752cf6d47b0c416 (127.30.218.2:39015): tablet 3e9c6aa3120b48168ddc7249a78d8153 (table test-table [id=13c2b7f7bd6c480fbf868fbb3897d4bb]) successfully deleted
I20260806 10:58:41.995913 20013 tablet_replica.cc:330] T 3e9c6aa3120b48168ddc7249a78d8153 P eb5ef90d8dc04825b62fa75e2ee4261d: stopping tablet replica
I20260806 10:58:41.996177 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 19564
W20260806 10:58:42.014046 19488 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.218.1:42511 (error 108)
I20260806 10:58:42.014484 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 19701
I20260806 10:58:42.016487 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:42.015833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:42511 (local address 127.0.0.1:53476)
0806 10:58:42.016090 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:42.016336 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260806 10:58:42.016770 19488 catalog_manager.cc:4584] TS eb5ef90d8dc04825b62fa75e2ee4261d (127.30.218.1:42511): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3e9c6aa3120b48168ddc7249a78d8153: Network error: Client connection negotiation failed: client connection to 127.30.218.1:42511: connect: Connection refused (error 111)
I20260806 10:58:42.023059 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:42.015354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38532 (local address 127.30.218.2:39015)
0806 10:58:42.015555 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:42.015561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:42.015582 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:58:42.015836 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:42.015843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:42.015967 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:42.016075 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:42.016752 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:42.017953 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:42.019332 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:42.020041 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:42.020176 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:42.021237 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:42.021250 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:42.021271 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:42.022180 (+ 909us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:42.022271 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:42.022298 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:42.022418 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:42.022526 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 10:58:42.022642 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 10:58:42.023996 19829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:42.070776 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:42.070195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:42511 (local address 127.0.0.1:53480)
0806 10:58:42.070454 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:42.070571 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260806 10:58:42.071880 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:42.071488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:42511 (local address 127.0.0.1:53482)
0806 10:58:42.071638 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:42.071727 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 10:58:42.132540 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:42.131899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:42511 (local address 127.0.0.1:53484)
0806 10:58:42.132185 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:42.132321 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260806 10:58:42.133760 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:42.133294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:42511 (local address 127.0.0.1:53498)
0806 10:58:42.133505 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:42.133606 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 10:58:42.140408 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 19701
I20260806 10:58:42.158864 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 19837
I20260806 10:58:42.165658 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:42.159633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52724 (local address 127.30.218.3:41883)
0806 10:58:42.159832 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:42.159836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:42.159855 (+ 19us) server_negotiation.cc:407] Connection header received
0806 10:58:42.159956 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:42.159961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:42.160042 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:42.160130 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:42.160574 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:42.161327 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:42.162624 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:42.163043 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:42.163135 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:42.164260 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:42.164270 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:42.164285 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:42.164879 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:42.164930 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:42.164950 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:42.165052 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:42.165196 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 10:58:42.165322 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 10:58:42.166446 19963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:42.214553 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:42.213953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:42511 (local address 127.0.0.1:53502)
0806 10:58:42.214201 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:42.214369 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260806 10:58:42.215674 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:42.215236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:42511 (local address 127.0.0.1:53514)
0806 10:58:42.215440 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:42.215536 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260806 10:58:42.256740 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 19837
I20260806 10:58:42.274896 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 19471
I20260806 10:58:42.276039 19533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:42.388797 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 19471
I20260806 10:58:42.388897 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:42.388311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.1:42511 (local address 127.0.0.1:53524)
0806 10:58:42.388558 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 10:58:42.388678 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.1:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
2026-08-06T10:58:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2863 ms)
[----------] 6 tests from EncryptionPolicyTest (23357 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:58:42 dist-test-slave-49h3 krb5kdc[20023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:58:42 dist-test-slave-49h3 krb5kdc[20023](info): set up 2 sockets
Aug 06 10:58:42 dist-test-slave-49h3 krb5kdc[20023](info): commencing operation
krb5kdc: starting...
W20260806 10:58:43.471381 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:58:43 dist-test-slave-49h3 krb5kdc[20023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:58:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:58:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 10:58:43.814103 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32949
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:58:44.200405 20036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:44.200793 20036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:44.200959 20036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:44.208877 20036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:58:44.209054 20036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:44.209147 20036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:44.209237 20036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:58:44.209304 20036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:58:44.220698 20036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32949
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:44.222486 20036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:44.224120 20036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:44.237818 20041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:44.237746 20042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:44.237794 20044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:44.239086 20036 server_base.cc:1034] running on GCE node
I20260806 10:58:44.240020 20036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:44.241815 20036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:44.243031 20036 hybrid_clock.cc:648] HybridClock initialized: now 1786013924242987 us; error 39 us; skew 500 ppm
I20260806 10:58:44.243407 20036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:58:44 dist-test-slave-49h3 krb5kdc[20023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:58:44.247015 20036 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 10:58:44.248822 20036 webserver.cc:460] Webserver started at http://127.0.0.1:41293/ using document root <none> and password file <none>
I20260806 10:58:44.249908 20036 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:44.250062 20036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:44.250407 20036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:44.254550 20036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "9831f47e033a443086d9910484c77c67"
format_stamp: "Formatted at 2026-08-06 10:58:44 on dist-test-slave-49h3"
server_key: "adf1706fa2fd86db6be69ce210b4bbd9"
server_key_iv: "5b88d7f3fea3762c1b8ec2efff1b99e1"
server_key_version: "encryptionkey@0"
I20260806 10:58:44.255664 20036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "9831f47e033a443086d9910484c77c67"
format_stamp: "Formatted at 2026-08-06 10:58:44 on dist-test-slave-49h3"
server_key: "adf1706fa2fd86db6be69ce210b4bbd9"
server_key_iv: "5b88d7f3fea3762c1b8ec2efff1b99e1"
server_key_version: "encryptionkey@0"
I20260806 10:58:44.262085 20036 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 10:58:44.266155 20052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:44.267768 20036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:58:44.267962 20036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "9831f47e033a443086d9910484c77c67"
format_stamp: "Formatted at 2026-08-06 10:58:44 on dist-test-slave-49h3"
server_key: "adf1706fa2fd86db6be69ce210b4bbd9"
server_key_iv: "5b88d7f3fea3762c1b8ec2efff1b99e1"
server_key_version: "encryptionkey@0"
I20260806 10:58:44.268177 20036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:44.296291 20036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:44.301004 20036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:44.301302 20036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:44.321506 20036 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32949
I20260806 10:58:44.321543 20106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32949 every 8 connection(s)
I20260806 10:58:44.323096 20036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:58:44.327594 20107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:44.328549 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 20036
I20260806 10:58:44.328812 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:58:44.329778 31592 external_mini_cluster.cc:1428] Setting key 87db5a4588d7acf141ccb6c83a9e91f3
I20260806 10:58:44.343644 20107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9831f47e033a443086d9910484c77c67: Bootstrap starting.
Aug 06 10:58:44 dist-test-slave-49h3 krb5kdc[20023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 10:58:44.348335 20107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9831f47e033a443086d9910484c77c67: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:44.349671 20107 log.cc:824] T 00000000000000000000000000000000 P 9831f47e033a443086d9910484c77c67: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:44.353801 20107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9831f47e033a443086d9910484c77c67: No bootstrap required, opened a new log
I20260806 10:58:44.357350 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:44.332081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53373 (local address 127.0.0.1:32949)
0806 10:58:44.332937 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:44.332956 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:44.333009 (+ 53us) server_negotiation.cc:407] Connection header received
0806 10:58:44.334869 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:44.334894 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:44.335341 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:44.335728 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:44.337684 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:44.339186 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:44.341221 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:44.342091 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:44.342248 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:44.348026 (+ 5778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:44.348036 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:44.348053 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:44.348108 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:44.352115 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:44.353222 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:44.353229 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:44.353237 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:44.353495 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:44.354075 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:44.354079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:44.354081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:44.354772 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:44.355059 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:44.355611 (+ 552us) server_negotiation.cc:299] Negotiation successful
0806 10:58:44.355968 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":382,"threads_started":1}
I20260806 10:58:44.362414 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:58:44.363823 20107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9831f47e033a443086d9910484c77c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9831f47e033a443086d9910484c77c67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32949 } }
I20260806 10:58:44.364259 20107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9831f47e033a443086d9910484c77c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:44.364372 20107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9831f47e033a443086d9910484c77c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9831f47e033a443086d9910484c77c67, State: Initialized, Role: FOLLOWER
I20260806 10:58:44.365049 20107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9831f47e033a443086d9910484c77c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9831f47e033a443086d9910484c77c67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32949 } }
I20260806 10:58:44.365319 20107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9831f47e033a443086d9910484c77c67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:58:44.365509 20107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9831f47e033a443086d9910484c77c67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:58:44.365721 20107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9831f47e033a443086d9910484c77c67 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:44.368484 20107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9831f47e033a443086d9910484c77c67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9831f47e033a443086d9910484c77c67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32949 } }
I20260806 10:58:44.369138 20107 leader_election.cc:302] T 00000000000000000000000000000000 P 9831f47e033a443086d9910484c77c67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9831f47e033a443086d9910484c77c67; no voters:
I20260806 10:58:44.369853 20107 leader_election.cc:288] T 00000000000000000000000000000000 P 9831f47e033a443086d9910484c77c67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:58:44.370008 20117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9831f47e033a443086d9910484c77c67 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:58:44.370812 20117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9831f47e033a443086d9910484c77c67 [term 1 LEADER]: Becoming Leader. State: Replica: 9831f47e033a443086d9910484c77c67, State: Running, Role: LEADER
I20260806 10:58:44.371605 20117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9831f47e033a443086d9910484c77c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9831f47e033a443086d9910484c77c67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32949 } }
I20260806 10:58:44.374965 20119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9831f47e033a443086d9910484c77c67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9831f47e033a443086d9910484c77c67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9831f47e033a443086d9910484c77c67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32949 } } }
I20260806 10:58:44.375464 20119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9831f47e033a443086d9910484c77c67 [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:44.376698 20118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9831f47e033a443086d9910484c77c67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9831f47e033a443086d9910484c77c67. Latest consensus state: current_term: 1 leader_uuid: "9831f47e033a443086d9910484c77c67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9831f47e033a443086d9910484c77c67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32949 } } }
I20260806 10:58:44.379510 20118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9831f47e033a443086d9910484c77c67 [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:44.379323 20107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9831f47e033a443086d9910484c77c67 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:58:44.386077 20121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:58:44.388363 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:44.368049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43621 (local address 127.0.0.1:32949)
0806 10:58:44.368250 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:44.368256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:44.368710 (+ 454us) server_negotiation.cc:407] Connection header received
0806 10:58:44.368901 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:44.368906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:44.368997 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:44.369104 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:44.371149 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:44.372413 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:44.374366 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:44.375100 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:44.375221 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:44.379268 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:44.379276 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:44.379282 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:44.379336 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:44.383667 (+ 4331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:44.384828 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:44.384835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:44.384839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:44.385018 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:44.385669 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:44.385675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:44.385678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:44.386186 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:44.386726 (+ 540us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:44.387834 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0806 10:58:44.388068 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 10:58:44.396176 20121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:58:44.407919 20121 catalog_manager.cc:1347] Generated new cluster ID: 3b67c18bca3a411682490e568a09ca78
I20260806 10:58:44.408090 20121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:58:44.465906 20121 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:58:44.467986 20121 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:58:44.481261 20121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9831f47e033a443086d9910484c77c67: Generated new TSK 0
I20260806 10:58:44.482424 20121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:58:44.498135 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 20036
I20260806 10:58:44.500432 20102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:44.605688 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 20036
W20260806 10:58:44.628379 20020 connection.cc:535] client connection to 127.0.0.1:32949 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32949): Connection reset by peer (error 104)
2026-08-06T10:58:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2212 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:58:44 dist-test-slave-49h3 krb5kdc[20140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:58:44 dist-test-slave-49h3 krb5kdc[20140](info): set up 2 sockets
Aug 06 10:58:44 dist-test-slave-49h3 krb5kdc[20140](info): commencing operation
krb5kdc: starting...
W20260806 10:58:45.719213 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.004s sys 0.062s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:58:45 dist-test-slave-49h3 krb5kdc[20140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:58:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:58:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 10:58:46.068617 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43129
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:58:46.436488 20154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:46.436810 20154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:46.436952 20154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:46.444299 20154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:58:46.444435 20154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:46.444519 20154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:46.444607 20154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:58:46.444666 20154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:58:46.455184 20154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43129
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:46.456647 20154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:46.458431 20154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:46.470443 20160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:46.470386 20162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:46.470878 20159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:46.471318 20154 server_base.cc:1034] running on GCE node
I20260806 10:58:46.472030 20154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:46.473357 20154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:46.474567 20154 hybrid_clock.cc:648] HybridClock initialized: now 1786013926474521 us; error 35 us; skew 500 ppm
I20260806 10:58:46.474911 20154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:58:46 dist-test-slave-49h3 krb5kdc[20140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:58:46.477957 20154 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 10:58:46.479496 20154 webserver.cc:460] Webserver started at http://127.0.0.1:36937/ using document root <none> and password file <none>
I20260806 10:58:46.480183 20154 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:46.480290 20154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:46.480573 20154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:46.483661 20154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "b657aa2172fa46eb86d104b47c9ddd57"
format_stamp: "Formatted at 2026-08-06 10:58:46 on dist-test-slave-49h3"
server_key: "7c0a1d2cae41af5897af9162d16cd4db"
server_key_iv: "c8f60da1b529f704d2e05577ead7797d"
server_key_version: "encryptionkey@0"
I20260806 10:58:46.484457 20154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "b657aa2172fa46eb86d104b47c9ddd57"
format_stamp: "Formatted at 2026-08-06 10:58:46 on dist-test-slave-49h3"
server_key: "7c0a1d2cae41af5897af9162d16cd4db"
server_key_iv: "c8f60da1b529f704d2e05577ead7797d"
server_key_version: "encryptionkey@0"
I20260806 10:58:46.489908 20154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260806 10:58:46.493369 20169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:46.494699 20154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:58:46.494860 20154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "b657aa2172fa46eb86d104b47c9ddd57"
format_stamp: "Formatted at 2026-08-06 10:58:46 on dist-test-slave-49h3"
server_key: "7c0a1d2cae41af5897af9162d16cd4db"
server_key_iv: "c8f60da1b529f704d2e05577ead7797d"
server_key_version: "encryptionkey@0"
I20260806 10:58:46.495039 20154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:46.510808 20154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:46.514829 20154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:46.515106 20154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:46.533792 20154 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43129
I20260806 10:58:46.533851 20222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43129 every 8 connection(s)
I20260806 10:58:46.535300 20154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:58:46.539496 20223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:46.542317 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 20154
I20260806 10:58:46.542505 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:58:46.543001 31592 external_mini_cluster.cc:1428] Setting key 56203706846b8572bd85bb48fb46fef1
I20260806 10:58:46.554288 20223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b657aa2172fa46eb86d104b47c9ddd57: Bootstrap starting.
I20260806 10:58:46.558799 20223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b657aa2172fa46eb86d104b47c9ddd57: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:46.560256 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:46.545201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45705 (local address 127.0.0.1:43129)
0806 10:58:46.546164 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:46.546181 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:46.546240 (+ 59us) server_negotiation.cc:407] Connection header received
0806 10:58:46.548719 (+ 2479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:46.548754 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:46.549384 (+ 630us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:46.549956 (+ 572us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:46.550609 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:46.552810 (+ 2201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:46.554431 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:46.555510 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:46.555705 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:46.557065 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:46.557077 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:46.557098 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:46.557912 (+ 814us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:46.558026 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:46.558062 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:46.558241 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:46.558488 (+ 247us) server_negotiation.cc:299] Negotiation successful
0806 10:58:46.558699 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":383,"threads_started":1}
I20260806 10:58:46.560387 20223 log.cc:824] T 00000000000000000000000000000000 P b657aa2172fa46eb86d104b47c9ddd57: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:46.564079 20223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b657aa2172fa46eb86d104b47c9ddd57: No bootstrap required, opened a new log
I20260806 10:58:46.565292 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:58:46.574990 20223 raft_consensus.cc:348] T 00000000000000000000000000000000 P b657aa2172fa46eb86d104b47c9ddd57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b657aa2172fa46eb86d104b47c9ddd57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43129 } }
I20260806 10:58:46.575558 20223 raft_consensus.cc:374] T 00000000000000000000000000000000 P b657aa2172fa46eb86d104b47c9ddd57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:46.575729 20223 raft_consensus.cc:729] T 00000000000000000000000000000000 P b657aa2172fa46eb86d104b47c9ddd57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b657aa2172fa46eb86d104b47c9ddd57, State: Initialized, Role: FOLLOWER
I20260806 10:58:46.576421 20223 consensus_queue.cc:260] T 00000000000000000000000000000000 P b657aa2172fa46eb86d104b47c9ddd57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b657aa2172fa46eb86d104b47c9ddd57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43129 } }
I20260806 10:58:46.576741 20223 raft_consensus.cc:388] T 00000000000000000000000000000000 P b657aa2172fa46eb86d104b47c9ddd57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:58:46.576897 20223 raft_consensus.cc:482] T 00000000000000000000000000000000 P b657aa2172fa46eb86d104b47c9ddd57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:58:46.577086 20223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b657aa2172fa46eb86d104b47c9ddd57 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:46.578501 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:46.571363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36603 (local address 127.0.0.1:43129)
0806 10:58:46.571591 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:46.571597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:46.571934 (+ 337us) server_negotiation.cc:407] Connection header received
0806 10:58:46.572140 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:46.572144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:46.572215 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:46.572316 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:46.572952 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:46.573835 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:46.575698 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:46.576282 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:46.576384 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:46.577702 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:46.577710 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:46.577713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:46.577740 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:46.577806 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:46.577828 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:46.577936 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:46.578137 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 10:58:46.578198 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 10:58:46.580137 20223 raft_consensus.cc:504] T 00000000000000000000000000000000 P b657aa2172fa46eb86d104b47c9ddd57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b657aa2172fa46eb86d104b47c9ddd57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43129 } }
I20260806 10:58:46.580730 20223 leader_election.cc:302] T 00000000000000000000000000000000 P b657aa2172fa46eb86d104b47c9ddd57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b657aa2172fa46eb86d104b47c9ddd57; no voters:
I20260806 10:58:46.581286 20223 leader_election.cc:288] T 00000000000000000000000000000000 P b657aa2172fa46eb86d104b47c9ddd57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:58:46.581437 20235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b657aa2172fa46eb86d104b47c9ddd57 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:58:46.582086 20235 raft_consensus.cc:686] T 00000000000000000000000000000000 P b657aa2172fa46eb86d104b47c9ddd57 [term 1 LEADER]: Becoming Leader. State: Replica: b657aa2172fa46eb86d104b47c9ddd57, State: Running, Role: LEADER
I20260806 10:58:46.582656 20235 consensus_queue.cc:237] T 00000000000000000000000000000000 P b657aa2172fa46eb86d104b47c9ddd57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b657aa2172fa46eb86d104b47c9ddd57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43129 } }
I20260806 10:58:46.583410 20223 sys_catalog.cc:564] T 00000000000000000000000000000000 P b657aa2172fa46eb86d104b47c9ddd57 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:58:46.585326 20236 sys_catalog.cc:455] T 00000000000000000000000000000000 P b657aa2172fa46eb86d104b47c9ddd57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b657aa2172fa46eb86d104b47c9ddd57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b657aa2172fa46eb86d104b47c9ddd57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43129 } } }
I20260806 10:58:46.585758 20236 sys_catalog.cc:458] T 00000000000000000000000000000000 P b657aa2172fa46eb86d104b47c9ddd57 [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:46.586701 20237 sys_catalog.cc:455] T 00000000000000000000000000000000 P b657aa2172fa46eb86d104b47c9ddd57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b657aa2172fa46eb86d104b47c9ddd57. Latest consensus state: current_term: 1 leader_uuid: "b657aa2172fa46eb86d104b47c9ddd57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b657aa2172fa46eb86d104b47c9ddd57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43129 } } }
I20260806 10:58:46.587143 20237 sys_catalog.cc:458] T 00000000000000000000000000000000 P b657aa2172fa46eb86d104b47c9ddd57 [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:46.587728 20240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:58:46.593560 20240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:58:46.604099 20240 catalog_manager.cc:1347] Generated new cluster ID: 7be7453a2a394ec6a10704a134fbd4b2
I20260806 10:58:46.604239 20240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:58:46.617878 20240 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:58:46.619241 20240 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:58:46.628374 20240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b657aa2172fa46eb86d104b47c9ddd57: Generated new TSK 0
I20260806 10:58:46.629235 20240 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:58:46.648170 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 20154
W20260806 10:58:46.649497 20218 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45705
W20260806 10:58:46.650377 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 20154: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:58:46.650564 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 20154
W20260806 10:58:46.668921 20138 connection.cc:535] client connection to 127.0.0.1:43129 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43129): Connection reset by peer (error 104)
2026-08-06T10:58:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2039 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:58:46 dist-test-slave-49h3 krb5kdc[20257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:58:46 dist-test-slave-49h3 krb5kdc[20257](info): set up 2 sockets
Aug 06 10:58:46 dist-test-slave-49h3 krb5kdc[20257](info): commencing operation
krb5kdc: starting...
W20260806 10:58:47.735122 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.003s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:58:47 dist-test-slave-49h3 krb5kdc[20257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:58:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:58:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 10:58:48.082854 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:58:48.436769 20270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:48.437134 20270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:48.437282 20270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:48.444967 20270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:58:48.445128 20270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:48.445220 20270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:48.445305 20270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:58:48.445365 20270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:58:48.455952 20270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38691
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:48.457373 20270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:48.458815 20270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:48.470772 20276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:48.470803 20278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:48.470886 20275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:48.471459 20270 server_base.cc:1034] running on GCE node
I20260806 10:58:48.472290 20270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:48.473753 20270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:48.474947 20270 hybrid_clock.cc:648] HybridClock initialized: now 1786013928474895 us; error 44 us; skew 500 ppm
I20260806 10:58:48.475322 20270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:58:48 dist-test-slave-49h3 krb5kdc[20257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:58:48.478490 20270 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 10:58:48.480013 20270 webserver.cc:460] Webserver started at http://127.0.0.1:37893/ using document root <none> and password file <none>
I20260806 10:58:48.480708 20270 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:48.480824 20270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:48.481130 20270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:48.484238 20270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "d1fdb71613044d6d8f3c71102d73f7d1"
format_stamp: "Formatted at 2026-08-06 10:58:48 on dist-test-slave-49h3"
server_key: "1dc2496a611e8d7980c9313fffa7526d"
server_key_iv: "96fb7decff996490be7e4e5b94b7f66f"
server_key_version: "encryptionkey@0"
I20260806 10:58:48.485035 20270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "d1fdb71613044d6d8f3c71102d73f7d1"
format_stamp: "Formatted at 2026-08-06 10:58:48 on dist-test-slave-49h3"
server_key: "1dc2496a611e8d7980c9313fffa7526d"
server_key_iv: "96fb7decff996490be7e4e5b94b7f66f"
server_key_version: "encryptionkey@0"
I20260806 10:58:48.490748 20270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 10:58:48.494493 20285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:48.495921 20270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:58:48.496102 20270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "d1fdb71613044d6d8f3c71102d73f7d1"
format_stamp: "Formatted at 2026-08-06 10:58:48 on dist-test-slave-49h3"
server_key: "1dc2496a611e8d7980c9313fffa7526d"
server_key_iv: "96fb7decff996490be7e4e5b94b7f66f"
server_key_version: "encryptionkey@0"
I20260806 10:58:48.496312 20270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:48.502349 20270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:48.507035 20270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:48.507355 20270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:48.526064 20270 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38691
I20260806 10:58:48.526191 20339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38691 every 8 connection(s)
I20260806 10:58:48.527526 20270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:58:48.531373 20340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:48.536747 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 20270
I20260806 10:58:48.536945 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:58:48.537546 31592 external_mini_cluster.cc:1428] Setting key 37e863404b34a753aae31b15d58d7847
I20260806 10:58:48.543190 20340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1fdb71613044d6d8f3c71102d73f7d1: Bootstrap starting.
I20260806 10:58:48.545812 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:48.539418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56131 (local address 127.0.0.1:38691)
0806 10:58:48.540278 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:48.540291 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:48.540341 (+ 50us) server_negotiation.cc:407] Connection header received
0806 10:58:48.542198 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:48.542224 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:48.542597 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:48.542940 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:48.544089 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:48.544097 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:48.544110 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:48.544127 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:48.544202 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:48.544225 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:48.544419 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:48.544533 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 10:58:48.544718 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":379,"threads_started":1}
I20260806 10:58:48.547199 20340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1fdb71613044d6d8f3c71102d73f7d1: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:48.548388 20340 log.cc:824] T 00000000000000000000000000000000 P d1fdb71613044d6d8f3c71102d73f7d1: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:48.550060 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:58:48.552157 20340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1fdb71613044d6d8f3c71102d73f7d1: No bootstrap required, opened a new log
I20260806 10:58:48.558144 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:48.555540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46099 (local address 127.0.0.1:38691)
0806 10:58:48.555749 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:48.555755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:48.556154 (+ 399us) server_negotiation.cc:407] Connection header received
0806 10:58:48.556323 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:48.556328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:48.556396 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:48.556532 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:48.557424 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:48.557430 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:48.557434 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:48.557484 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:48.557542 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:48.557575 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:48.557719 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:48.557764 (+ 45us) server_negotiation.cc:299] Negotiation successful
0806 10:58:48.557830 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 10:58:48.561975 20340 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1fdb71613044d6d8f3c71102d73f7d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1fdb71613044d6d8f3c71102d73f7d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38691 } }
I20260806 10:58:48.562402 20340 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1fdb71613044d6d8f3c71102d73f7d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:48.562482 20340 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1fdb71613044d6d8f3c71102d73f7d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1fdb71613044d6d8f3c71102d73f7d1, State: Initialized, Role: FOLLOWER
I20260806 10:58:48.563022 20340 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1fdb71613044d6d8f3c71102d73f7d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1fdb71613044d6d8f3c71102d73f7d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38691 } }
I20260806 10:58:48.563283 20340 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1fdb71613044d6d8f3c71102d73f7d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:58:48.563392 20340 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1fdb71613044d6d8f3c71102d73f7d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:58:48.563542 20340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1fdb71613044d6d8f3c71102d73f7d1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:48.566437 20340 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1fdb71613044d6d8f3c71102d73f7d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1fdb71613044d6d8f3c71102d73f7d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38691 } }
I20260806 10:58:48.567000 20340 leader_election.cc:302] T 00000000000000000000000000000000 P d1fdb71613044d6d8f3c71102d73f7d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1fdb71613044d6d8f3c71102d73f7d1; no voters:
I20260806 10:58:48.567611 20340 leader_election.cc:288] T 00000000000000000000000000000000 P d1fdb71613044d6d8f3c71102d73f7d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:58:48.567764 20350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1fdb71613044d6d8f3c71102d73f7d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:58:48.568406 20350 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1fdb71613044d6d8f3c71102d73f7d1 [term 1 LEADER]: Becoming Leader. State: Replica: d1fdb71613044d6d8f3c71102d73f7d1, State: Running, Role: LEADER
I20260806 10:58:48.569013 20350 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1fdb71613044d6d8f3c71102d73f7d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1fdb71613044d6d8f3c71102d73f7d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38691 } }
I20260806 10:58:48.569708 20340 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1fdb71613044d6d8f3c71102d73f7d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:58:48.572587 20352 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1fdb71613044d6d8f3c71102d73f7d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1fdb71613044d6d8f3c71102d73f7d1. Latest consensus state: current_term: 1 leader_uuid: "d1fdb71613044d6d8f3c71102d73f7d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1fdb71613044d6d8f3c71102d73f7d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38691 } } }
I20260806 10:58:48.572639 20351 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1fdb71613044d6d8f3c71102d73f7d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1fdb71613044d6d8f3c71102d73f7d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1fdb71613044d6d8f3c71102d73f7d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38691 } } }
I20260806 10:58:48.573166 20352 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1fdb71613044d6d8f3c71102d73f7d1 [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:48.573287 20351 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1fdb71613044d6d8f3c71102d73f7d1 [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:48.574215 20359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:58:48.580531 20359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:58:48.590415 20359 catalog_manager.cc:1347] Generated new cluster ID: c18576a62f76455da4b4210cad2938dc
I20260806 10:58:48.590565 20359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:58:48.610227 20359 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:58:48.610857 20359 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:58:48.621534 20359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1fdb71613044d6d8f3c71102d73f7d1: Generated new TSK 0
I20260806 10:58:48.622483 20359 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:58:48.631699 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 20270
W20260806 10:58:48.632951 20335 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56131
W20260806 10:58:48.633706 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 20270: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:58:48.633849 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 20270
2026-08-06T10:58:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1982 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:58:48 dist-test-slave-49h3 krb5kdc[20372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:58:48 dist-test-slave-49h3 krb5kdc[20372](info): set up 2 sockets
Aug 06 10:58:48 dist-test-slave-49h3 krb5kdc[20372](info): commencing operation
krb5kdc: starting...
W20260806 10:58:49.719203 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.001s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:58:49 dist-test-slave-49h3 krb5kdc[20372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:58:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:58:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 10:58:50.062209 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:58:50.414737 20385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:50.415086 20385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:50.415228 20385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:50.422763 20385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:58:50.422900 20385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:50.422973 20385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:50.423082 20385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:58:50.423141 20385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:58:50.433550 20385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:50.434928 20385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:50.436261 20385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:50.448125 20393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:50.448072 20390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:50.448071 20391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:50.449321 20385 server_base.cc:1034] running on GCE node
I20260806 10:58:50.450270 20385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:50.451858 20385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:50.453110 20385 hybrid_clock.cc:648] HybridClock initialized: now 1786013930453056 us; error 54 us; skew 500 ppm
I20260806 10:58:50.453545 20385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:58:50 dist-test-slave-49h3 krb5kdc[20372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:58:50.456902 20385 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 10:58:50.458619 20385 webserver.cc:460] Webserver started at http://127.0.0.1:39783/ using document root <none> and password file <none>
I20260806 10:58:50.459342 20385 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:50.459458 20385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:50.459736 20385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:50.463017 20385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "655090ad1e32456a827b36ecca6c44af"
format_stamp: "Formatted at 2026-08-06 10:58:50 on dist-test-slave-49h3"
server_key: "11fbf4c371a76579671cfe6a03c128d4"
server_key_iv: "4a72b2a414c77ac769a7aa627a2ff3ad"
server_key_version: "encryptionkey@0"
I20260806 10:58:50.463824 20385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "655090ad1e32456a827b36ecca6c44af"
format_stamp: "Formatted at 2026-08-06 10:58:50 on dist-test-slave-49h3"
server_key: "11fbf4c371a76579671cfe6a03c128d4"
server_key_iv: "4a72b2a414c77ac769a7aa627a2ff3ad"
server_key_version: "encryptionkey@0"
I20260806 10:58:50.469436 20385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 10:58:50.473193 20401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:50.474619 20385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:58:50.474789 20385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "655090ad1e32456a827b36ecca6c44af"
format_stamp: "Formatted at 2026-08-06 10:58:50 on dist-test-slave-49h3"
server_key: "11fbf4c371a76579671cfe6a03c128d4"
server_key_iv: "4a72b2a414c77ac769a7aa627a2ff3ad"
server_key_version: "encryptionkey@0"
I20260806 10:58:50.474975 20385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:50.497344 20385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:50.501994 20385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:50.502267 20385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:50.521306 20385 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45889
I20260806 10:58:50.521358 20453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45889 every 8 connection(s)
I20260806 10:58:50.522861 20385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:58:50.526939 20454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:50.527833 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 20385
I20260806 10:58:50.528062 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:58:50.528649 31592 external_mini_cluster.cc:1428] Setting key 3bd1dee95b8d4f534d36d44029eb02fe
I20260806 10:58:50.541484 20454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 655090ad1e32456a827b36ecca6c44af: Bootstrap starting.
Aug 06 10:58:50 dist-test-slave-49h3 krb5kdc[20372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 10:58:50.545632 20454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 655090ad1e32456a827b36ecca6c44af: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:50.546841 20454 log.cc:824] T 00000000000000000000000000000000 P 655090ad1e32456a827b36ecca6c44af: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:50.550413 20454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 655090ad1e32456a827b36ecca6c44af: No bootstrap required, opened a new log
I20260806 10:58:50.554643 20457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:50.530648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43429 (local address 127.0.0.1:45889)
0806 10:58:50.531546 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:50.531562 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:50.531609 (+ 47us) server_negotiation.cc:407] Connection header received
0806 10:58:50.533573 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:50.533602 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:50.534074 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:50.534440 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:50.536320 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:50.537737 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:50.539509 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:50.540216 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:50.540350 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:50.545576 (+ 5226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:50.545586 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:50.545603 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:50.545654 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:50.549324 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:50.550336 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:50.550345 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:50.550353 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:50.550644 (+ 291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:50.551060 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:50.551066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:50.551069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:50.551833 (+ 764us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:50.552145 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:50.552590 (+ 445us) server_negotiation.cc:299] Negotiation successful
0806 10:58:50.553134 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":424,"threads_started":1}
I20260806 10:58:50.558872 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:58:50.560245 20454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 655090ad1e32456a827b36ecca6c44af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655090ad1e32456a827b36ecca6c44af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45889 } }
I20260806 10:58:50.560745 20454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 655090ad1e32456a827b36ecca6c44af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:50.560849 20454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 655090ad1e32456a827b36ecca6c44af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655090ad1e32456a827b36ecca6c44af, State: Initialized, Role: FOLLOWER
I20260806 10:58:50.561582 20454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 655090ad1e32456a827b36ecca6c44af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655090ad1e32456a827b36ecca6c44af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45889 } }
I20260806 10:58:50.561901 20454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 655090ad1e32456a827b36ecca6c44af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:58:50.562048 20454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 655090ad1e32456a827b36ecca6c44af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:58:50.562242 20454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 655090ad1e32456a827b36ecca6c44af [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:50.565332 20454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 655090ad1e32456a827b36ecca6c44af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655090ad1e32456a827b36ecca6c44af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45889 } }
I20260806 10:58:50.565975 20454 leader_election.cc:302] T 00000000000000000000000000000000 P 655090ad1e32456a827b36ecca6c44af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 655090ad1e32456a827b36ecca6c44af; no voters:
I20260806 10:58:50.566632 20454 leader_election.cc:288] T 00000000000000000000000000000000 P 655090ad1e32456a827b36ecca6c44af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:58:50.566735 20464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 655090ad1e32456a827b36ecca6c44af [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:58:50.569185 20454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 655090ad1e32456a827b36ecca6c44af [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:58:50.573400 20464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 655090ad1e32456a827b36ecca6c44af [term 1 LEADER]: Becoming Leader. State: Replica: 655090ad1e32456a827b36ecca6c44af, State: Running, Role: LEADER
I20260806 10:58:50.574335 20464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 655090ad1e32456a827b36ecca6c44af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655090ad1e32456a827b36ecca6c44af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45889 } }
I20260806 10:58:50.577827 20465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 655090ad1e32456a827b36ecca6c44af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "655090ad1e32456a827b36ecca6c44af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655090ad1e32456a827b36ecca6c44af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45889 } } }
I20260806 10:58:50.578356 20465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 655090ad1e32456a827b36ecca6c44af [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:50.580794 20457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:50.564478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49283 (local address 127.0.0.1:45889)
0806 10:58:50.564685 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:50.564692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:50.565064 (+ 372us) server_negotiation.cc:407] Connection header received
0806 10:58:50.565216 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:50.565222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:50.565309 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:50.565436 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:50.567147 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:50.568153 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:50.569536 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:50.570031 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:50.570167 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:50.572804 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:50.572809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:50.572813 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:50.572849 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:50.575958 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:50.576900 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:50.576907 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:50.576910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:50.577102 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:50.577657 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:50.577662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:50.577665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:50.578121 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:50.578415 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:50.579880 (+ 1465us) server_negotiation.cc:299] Negotiation successful
0806 10:58:50.580190 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 10:58:50.584991 20466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 655090ad1e32456a827b36ecca6c44af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 655090ad1e32456a827b36ecca6c44af. Latest consensus state: current_term: 1 leader_uuid: "655090ad1e32456a827b36ecca6c44af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655090ad1e32456a827b36ecca6c44af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45889 } } }
I20260806 10:58:50.585685 20466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 655090ad1e32456a827b36ecca6c44af [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:50.588598 20472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:58:50.594553 20472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:58:50.605050 20472 catalog_manager.cc:1347] Generated new cluster ID: 71d4fa36549e4d3390b270bfaba84d10
I20260806 10:58:50.605196 20472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:58:50.625638 20472 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:58:50.627645 20472 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:58:50.638113 20472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 655090ad1e32456a827b36ecca6c44af: Generated new TSK 0
I20260806 10:58:50.639294 20472 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:58:50.652009 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 20385
I20260806 10:58:50.653335 20449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:50.750216 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 20385
2026-08-06T10:58:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2117 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:58:50 dist-test-slave-49h3 krb5kdc[20488](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:58:50 dist-test-slave-49h3 krb5kdc[20488](info): set up 2 sockets
Aug 06 10:58:50 dist-test-slave-49h3 krb5kdc[20488](info): commencing operation
krb5kdc: starting...
W20260806 10:58:51.842398 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:58:52 dist-test-slave-49h3 krb5kdc[20488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:58:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:58:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 10:58:52.194972 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43643
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:58:52.546857 20501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:52.547187 20501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:52.547322 20501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:52.554811 20501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:58:52.554951 20501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:52.555022 20501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:52.555109 20501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:58:52.555173 20501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:58:52.565578 20501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:52.566960 20501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:52.568281 20501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:52.579953 20509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:52.579998 20506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:52.579918 20507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:52.580610 20501 server_base.cc:1034] running on GCE node
I20260806 10:58:52.581472 20501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:52.582969 20501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:52.584182 20501 hybrid_clock.cc:648] HybridClock initialized: now 1786013932584130 us; error 49 us; skew 500 ppm
I20260806 10:58:52.584532 20501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:58:52 dist-test-slave-49h3 krb5kdc[20488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:58:52.592819 20501 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 10:58:52.594600 20501 webserver.cc:460] Webserver started at http://127.0.0.1:35301/ using document root <none> and password file <none>
I20260806 10:58:52.595353 20501 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:52.595492 20501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:52.595820 20501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:52.599226 20501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "b81bc3dbbb354558ab6ac2309259fccf"
format_stamp: "Formatted at 2026-08-06 10:58:52 on dist-test-slave-49h3"
server_key: "076f05552e928617461c42acc60b5e4c"
server_key_iv: "80e088a8306031fe92d31552e822caa3"
server_key_version: "encryptionkey@0"
I20260806 10:58:52.600040 20501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "b81bc3dbbb354558ab6ac2309259fccf"
format_stamp: "Formatted at 2026-08-06 10:58:52 on dist-test-slave-49h3"
server_key: "076f05552e928617461c42acc60b5e4c"
server_key_iv: "80e088a8306031fe92d31552e822caa3"
server_key_version: "encryptionkey@0"
I20260806 10:58:52.605582 20501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 10:58:52.609344 20517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:52.610864 20501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 10:58:52.611056 20501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "b81bc3dbbb354558ab6ac2309259fccf"
format_stamp: "Formatted at 2026-08-06 10:58:52 on dist-test-slave-49h3"
server_key: "076f05552e928617461c42acc60b5e4c"
server_key_iv: "80e088a8306031fe92d31552e822caa3"
server_key_version: "encryptionkey@0"
I20260806 10:58:52.611277 20501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:52.635599 20501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:52.640691 20501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:52.641009 20501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:52.660435 20501 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43643
I20260806 10:58:52.660467 20570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43643 every 8 connection(s)
I20260806 10:58:52.661953 20501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:58:52.665833 20571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:52.668757 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 20501
I20260806 10:58:52.668962 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:58:52.669543 31592 external_mini_cluster.cc:1428] Setting key 2d452f7f04b8ac3d6c366886ec217466
I20260806 10:58:52.679107 20571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b81bc3dbbb354558ab6ac2309259fccf: Bootstrap starting.
I20260806 10:58:52.683110 20571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b81bc3dbbb354558ab6ac2309259fccf: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:52.684019 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:52.671631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38919 (local address 127.0.0.1:43643)
0806 10:58:52.672497 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:52.672510 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:52.672555 (+ 45us) server_negotiation.cc:407] Connection header received
0806 10:58:52.674342 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:52.674369 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:52.674820 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:52.675205 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:52.675928 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:52.677339 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:52.679242 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:52.680009 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:52.680155 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:52.681531 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:52.681542 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:52.681559 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:52.682146 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:52.682229 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:52.682257 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:52.682347 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:52.682580 (+ 233us) server_negotiation.cc:299] Negotiation successful
0806 10:58:52.682853 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":388,"threads_started":1}
I20260806 10:58:52.684294 20571 log.cc:824] T 00000000000000000000000000000000 P b81bc3dbbb354558ab6ac2309259fccf: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:52.687562 20571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b81bc3dbbb354558ab6ac2309259fccf: No bootstrap required, opened a new log
I20260806 10:58:52.688234 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:58:52.698130 20571 raft_consensus.cc:348] T 00000000000000000000000000000000 P b81bc3dbbb354558ab6ac2309259fccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b81bc3dbbb354558ab6ac2309259fccf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43643 } }
I20260806 10:58:52.698660 20571 raft_consensus.cc:374] T 00000000000000000000000000000000 P b81bc3dbbb354558ab6ac2309259fccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:52.698781 20571 raft_consensus.cc:729] T 00000000000000000000000000000000 P b81bc3dbbb354558ab6ac2309259fccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81bc3dbbb354558ab6ac2309259fccf, State: Initialized, Role: FOLLOWER
I20260806 10:58:52.699481 20571 consensus_queue.cc:260] T 00000000000000000000000000000000 P b81bc3dbbb354558ab6ac2309259fccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b81bc3dbbb354558ab6ac2309259fccf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43643 } }
I20260806 10:58:52.699740 20571 raft_consensus.cc:388] T 00000000000000000000000000000000 P b81bc3dbbb354558ab6ac2309259fccf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:58:52.699895 20571 raft_consensus.cc:482] T 00000000000000000000000000000000 P b81bc3dbbb354558ab6ac2309259fccf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:58:52.700102 20571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b81bc3dbbb354558ab6ac2309259fccf [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:52.700455 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:52.693765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54779 (local address 127.0.0.1:43643)
0806 10:58:52.693930 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:52.693934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:52.694267 (+ 333us) server_negotiation.cc:407] Connection header received
0806 10:58:52.694455 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:52.694459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:52.694527 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:52.694624 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:52.695264 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:52.696021 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:52.697866 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:52.698334 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:52.698446 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:52.699660 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:52.699665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:52.699668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:52.699693 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:52.699752 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:52.699773 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:52.699861 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:52.700034 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 10:58:52.700131 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 10:58:52.702837 20571 raft_consensus.cc:504] T 00000000000000000000000000000000 P b81bc3dbbb354558ab6ac2309259fccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b81bc3dbbb354558ab6ac2309259fccf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43643 } }
I20260806 10:58:52.703389 20571 leader_election.cc:302] T 00000000000000000000000000000000 P b81bc3dbbb354558ab6ac2309259fccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b81bc3dbbb354558ab6ac2309259fccf; no voters:
I20260806 10:58:52.703965 20571 leader_election.cc:288] T 00000000000000000000000000000000 P b81bc3dbbb354558ab6ac2309259fccf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:58:52.704088 20581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b81bc3dbbb354558ab6ac2309259fccf [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:58:52.704763 20581 raft_consensus.cc:686] T 00000000000000000000000000000000 P b81bc3dbbb354558ab6ac2309259fccf [term 1 LEADER]: Becoming Leader. State: Replica: b81bc3dbbb354558ab6ac2309259fccf, State: Running, Role: LEADER
I20260806 10:58:52.705360 20581 consensus_queue.cc:237] T 00000000000000000000000000000000 P b81bc3dbbb354558ab6ac2309259fccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b81bc3dbbb354558ab6ac2309259fccf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43643 } }
I20260806 10:58:52.706172 20571 sys_catalog.cc:564] T 00000000000000000000000000000000 P b81bc3dbbb354558ab6ac2309259fccf [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:58:52.708267 20583 sys_catalog.cc:455] T 00000000000000000000000000000000 P b81bc3dbbb354558ab6ac2309259fccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b81bc3dbbb354558ab6ac2309259fccf. Latest consensus state: current_term: 1 leader_uuid: "b81bc3dbbb354558ab6ac2309259fccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b81bc3dbbb354558ab6ac2309259fccf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43643 } } }
I20260806 10:58:52.708691 20583 sys_catalog.cc:458] T 00000000000000000000000000000000 P b81bc3dbbb354558ab6ac2309259fccf [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:52.709622 20589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:58:52.710083 20582 sys_catalog.cc:455] T 00000000000000000000000000000000 P b81bc3dbbb354558ab6ac2309259fccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b81bc3dbbb354558ab6ac2309259fccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b81bc3dbbb354558ab6ac2309259fccf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43643 } } }
I20260806 10:58:52.710472 20582 sys_catalog.cc:458] T 00000000000000000000000000000000 P b81bc3dbbb354558ab6ac2309259fccf [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:52.716639 20589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:58:52.726135 20589 catalog_manager.cc:1347] Generated new cluster ID: 1324780b9dec41e5b76312d6aa95b862
I20260806 10:58:52.726300 20589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:58:52.747473 20589 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:58:52.748911 20589 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:58:52.755898 20589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b81bc3dbbb354558ab6ac2309259fccf: Generated new TSK 0
I20260806 10:58:52.756847 20589 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:58:52.768817 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 20501
W20260806 10:58:52.770040 20566 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38919
W20260806 10:58:52.770812 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 20501: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:58:52.770956 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 20501
2026-08-06T10:58:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2018 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:58:52 dist-test-slave-49h3 krb5kdc[20602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:58:52 dist-test-slave-49h3 krb5kdc[20602](info): set up 2 sockets
Aug 06 10:58:52 dist-test-slave-49h3 krb5kdc[20602](info): commencing operation
krb5kdc: starting...
W20260806 10:58:53.854710 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.002s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:58:54 dist-test-slave-49h3 krb5kdc[20602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:58:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:58:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 10:58:54.216614 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36109
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:58:54.588788 20615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:54.589114 20615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:54.589254 20615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:54.596696 20615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:58:54.596829 20615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:54.596906 20615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:54.596994 20615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:58:54.597060 20615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:58:54.607685 20615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:54.609128 20615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:54.610606 20615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:54.623313 20620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:54.623289 20623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:54.623374 20621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:54.623961 20615 server_base.cc:1034] running on GCE node
I20260806 10:58:54.625020 20615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:54.626734 20615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:54.627970 20615 hybrid_clock.cc:648] HybridClock initialized: now 1786013934627922 us; error 46 us; skew 500 ppm
I20260806 10:58:54.628356 20615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:58:54 dist-test-slave-49h3 krb5kdc[20602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:58:54.631862 20615 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 10:58:54.633601 20615 webserver.cc:460] Webserver started at http://127.0.0.1:38281/ using document root <none> and password file <none>
I20260806 10:58:54.634356 20615 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:54.634490 20615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:54.634846 20615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:54.638266 20615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "9d2acdf44e084bfaa3ef228169492469"
format_stamp: "Formatted at 2026-08-06 10:58:54 on dist-test-slave-49h3"
server_key: "cce8af8fc065046ec74fb74d2391b849"
server_key_iv: "a6096090c294956afdcc3782cd291dde"
server_key_version: "encryptionkey@0"
I20260806 10:58:54.639155 20615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "9d2acdf44e084bfaa3ef228169492469"
format_stamp: "Formatted at 2026-08-06 10:58:54 on dist-test-slave-49h3"
server_key: "cce8af8fc065046ec74fb74d2391b849"
server_key_iv: "a6096090c294956afdcc3782cd291dde"
server_key_version: "encryptionkey@0"
I20260806 10:58:54.645155 20615 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 10:58:54.649303 20630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:54.650907 20615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:58:54.651121 20615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "9d2acdf44e084bfaa3ef228169492469"
format_stamp: "Formatted at 2026-08-06 10:58:54 on dist-test-slave-49h3"
server_key: "cce8af8fc065046ec74fb74d2391b849"
server_key_iv: "a6096090c294956afdcc3782cd291dde"
server_key_version: "encryptionkey@0"
I20260806 10:58:54.651346 20615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:54.657979 20615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:54.662573 20615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:54.662915 20615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:54.681977 20615 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36109
I20260806 10:58:54.682036 20683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36109 every 8 connection(s)
I20260806 10:58:54.683672 20615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:58:54.687748 20685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:54.692684 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 20615
I20260806 10:58:54.692888 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:58:54.693506 31592 external_mini_cluster.cc:1428] Setting key e6c285a5ea4f2e44ed659d6709bb9263
I20260806 10:58:54.700214 20685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2acdf44e084bfaa3ef228169492469: Bootstrap starting.
I20260806 10:58:54.702328 20688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:54.695629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33359 (local address 127.0.0.1:36109)
0806 10:58:54.696486 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:54.696498 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:54.696548 (+ 50us) server_negotiation.cc:407] Connection header received
0806 10:58:54.698682 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:54.698709 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:54.699111 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:54.699494 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:54.700637 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:54.700643 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:54.700656 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:54.700672 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:54.700743 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:54.700778 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:54.700985 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:54.701126 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 10:58:54.701317 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":376,"threads_started":1}
I20260806 10:58:54.703701 20685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d2acdf44e084bfaa3ef228169492469: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:54.704891 20685 log.cc:824] T 00000000000000000000000000000000 P 9d2acdf44e084bfaa3ef228169492469: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:54.706367 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:58:54.708003 20685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2acdf44e084bfaa3ef228169492469: No bootstrap required, opened a new log
I20260806 10:58:54.713577 20688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:54.710882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48697 (local address 127.0.0.1:36109)
0806 10:58:54.711050 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:54.711055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:54.711518 (+ 463us) server_negotiation.cc:407] Connection header received
0806 10:58:54.711698 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:54.711707 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:54.711768 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:54.711866 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:54.712823 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:54.712829 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:54.712832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:54.712851 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:54.712892 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:54.712908 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:54.713020 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:54.713162 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 10:58:54.713238 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 10:58:54.716851 20685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d2acdf44e084bfaa3ef228169492469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2acdf44e084bfaa3ef228169492469" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36109 } }
I20260806 10:58:54.717326 20685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d2acdf44e084bfaa3ef228169492469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:54.717430 20685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d2acdf44e084bfaa3ef228169492469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2acdf44e084bfaa3ef228169492469, State: Initialized, Role: FOLLOWER
I20260806 10:58:54.718061 20685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d2acdf44e084bfaa3ef228169492469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2acdf44e084bfaa3ef228169492469" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36109 } }
I20260806 10:58:54.718299 20685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d2acdf44e084bfaa3ef228169492469 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:58:54.718451 20685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d2acdf44e084bfaa3ef228169492469 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:58:54.718613 20685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d2acdf44e084bfaa3ef228169492469 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:54.720930 20685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d2acdf44e084bfaa3ef228169492469 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2acdf44e084bfaa3ef228169492469" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36109 } }
I20260806 10:58:54.721478 20685 leader_election.cc:302] T 00000000000000000000000000000000 P 9d2acdf44e084bfaa3ef228169492469 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d2acdf44e084bfaa3ef228169492469; no voters:
I20260806 10:58:54.722105 20685 leader_election.cc:288] T 00000000000000000000000000000000 P 9d2acdf44e084bfaa3ef228169492469 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:58:54.722235 20695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d2acdf44e084bfaa3ef228169492469 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:58:54.723387 20695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d2acdf44e084bfaa3ef228169492469 [term 1 LEADER]: Becoming Leader. State: Replica: 9d2acdf44e084bfaa3ef228169492469, State: Running, Role: LEADER
I20260806 10:58:54.723985 20695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d2acdf44e084bfaa3ef228169492469 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2acdf44e084bfaa3ef228169492469" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36109 } }
I20260806 10:58:54.724346 20685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d2acdf44e084bfaa3ef228169492469 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:58:54.726608 20696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2acdf44e084bfaa3ef228169492469 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d2acdf44e084bfaa3ef228169492469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2acdf44e084bfaa3ef228169492469" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36109 } } }
I20260806 10:58:54.727092 20696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2acdf44e084bfaa3ef228169492469 [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:54.727632 20697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2acdf44e084bfaa3ef228169492469 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d2acdf44e084bfaa3ef228169492469. Latest consensus state: current_term: 1 leader_uuid: "9d2acdf44e084bfaa3ef228169492469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2acdf44e084bfaa3ef228169492469" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36109 } } }
I20260806 10:58:54.727908 20697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2acdf44e084bfaa3ef228169492469 [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:54.728145 20704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:58:54.736984 20704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:58:54.747802 20704 catalog_manager.cc:1347] Generated new cluster ID: 39b33a8465434e42a7b1c24325ad6108
I20260806 10:58:54.747949 20704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:58:54.761590 20704 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:58:54.762187 20704 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:58:54.785131 20704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d2acdf44e084bfaa3ef228169492469: Generated new TSK 0
I20260806 10:58:54.786434 20704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:58:54.794318 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 20615
W20260806 10:58:54.795827 20678 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33359
W20260806 10:58:54.796553 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 20615: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:58:54.796696 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 20615
2026-08-06T10:58:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2027 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:58:54 dist-test-slave-49h3 krb5kdc[20718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:58:54 dist-test-slave-49h3 krb5kdc[20718](info): set up 2 sockets
Aug 06 10:58:54 dist-test-slave-49h3 krb5kdc[20718](info): commencing operation
krb5kdc: starting...
W20260806 10:58:55.887261 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:58:56 dist-test-slave-49h3 krb5kdc[20718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:58:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:58:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 10:58:56.245496 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37879
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:58:56.612234 20731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:56.612567 20731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:56.612699 20731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:56.620379 20731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:58:56.620549 20731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:56.620718 20731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:56.620802 20731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:58:56.620870 20731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:58:56.631531 20731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:56.633004 20731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:56.634508 20731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:56.647415 20736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:56.647420 20737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:56.647373 20739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:56.651156 20731 server_base.cc:1034] running on GCE node
I20260806 10:58:56.652185 20731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:56.653873 20731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:56.655097 20731 hybrid_clock.cc:648] HybridClock initialized: now 1786013936655037 us; error 52 us; skew 500 ppm
I20260806 10:58:56.655488 20731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:58:56 dist-test-slave-49h3 krb5kdc[20718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:58:56.658977 20731 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 10:58:56.660759 20731 webserver.cc:460] Webserver started at http://127.0.0.1:41231/ using document root <none> and password file <none>
I20260806 10:58:56.661603 20731 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:56.661747 20731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:56.662076 20731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:56.665488 20731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "9fcd8c2376434248b9bbbd214127b9a8"
format_stamp: "Formatted at 2026-08-06 10:58:56 on dist-test-slave-49h3"
server_key: "92b10262b416c05ec315f68d7e810c89"
server_key_iv: "1c89b02d20a9ad3d043337eb6b0f79a8"
server_key_version: "encryptionkey@0"
I20260806 10:58:56.666455 20731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "9fcd8c2376434248b9bbbd214127b9a8"
format_stamp: "Formatted at 2026-08-06 10:58:56 on dist-test-slave-49h3"
server_key: "92b10262b416c05ec315f68d7e810c89"
server_key_iv: "1c89b02d20a9ad3d043337eb6b0f79a8"
server_key_version: "encryptionkey@0"
I20260806 10:58:56.672385 20731 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260806 10:58:56.676514 20746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:56.678064 20731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:58:56.678277 20731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "9fcd8c2376434248b9bbbd214127b9a8"
format_stamp: "Formatted at 2026-08-06 10:58:56 on dist-test-slave-49h3"
server_key: "92b10262b416c05ec315f68d7e810c89"
server_key_iv: "1c89b02d20a9ad3d043337eb6b0f79a8"
server_key_version: "encryptionkey@0"
I20260806 10:58:56.678494 20731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:56.700775 20731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:56.705899 20731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:56.706236 20731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:56.725689 20731 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37879
I20260806 10:58:56.725695 20797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37879 every 8 connection(s)
I20260806 10:58:56.727298 20731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:58:56.731994 20798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:56.733659 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 20731
I20260806 10:58:56.733865 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:58:56.734475 31592 external_mini_cluster.cc:1428] Setting key b89b28489e3cea74e93fdca754ab26a3
I20260806 10:58:56.746675 20798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fcd8c2376434248b9bbbd214127b9a8: Bootstrap starting.
Aug 06 10:58:56 dist-test-slave-49h3 krb5kdc[20718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 10:58:56.750941 20798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fcd8c2376434248b9bbbd214127b9a8: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:56.752197 20798 log.cc:824] T 00000000000000000000000000000000 P 9fcd8c2376434248b9bbbd214127b9a8: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:56.755425 20798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fcd8c2376434248b9bbbd214127b9a8: No bootstrap required, opened a new log
I20260806 10:58:56.760195 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:56.736760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:47433 (local address 127.0.0.1:37879)
0806 10:58:56.737744 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:56.737759 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:56.737812 (+ 53us) server_negotiation.cc:407] Connection header received
0806 10:58:56.739543 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:56.739576 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:56.740074 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:56.740510 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:56.742258 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:56.743704 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:56.745133 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:56.745957 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:56.746163 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:56.751329 (+ 5166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:56.751345 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:56.751363 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:56.751424 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:56.754715 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:56.755705 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:56.755715 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:56.755724 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:56.755971 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:56.756470 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:56.756475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:56.756478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:56.757261 (+ 783us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:56.757659 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:56.758143 (+ 484us) server_negotiation.cc:299] Negotiation successful
0806 10:58:56.758568 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":433,"threads_started":1}
I20260806 10:58:56.765147 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:58:56.765655 20798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fcd8c2376434248b9bbbd214127b9a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fcd8c2376434248b9bbbd214127b9a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } }
I20260806 10:58:56.766245 20798 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fcd8c2376434248b9bbbd214127b9a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:56.766355 20798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fcd8c2376434248b9bbbd214127b9a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fcd8c2376434248b9bbbd214127b9a8, State: Initialized, Role: FOLLOWER
I20260806 10:58:56.767112 20798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fcd8c2376434248b9bbbd214127b9a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fcd8c2376434248b9bbbd214127b9a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } }
I20260806 10:58:56.767418 20798 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fcd8c2376434248b9bbbd214127b9a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:58:56.767571 20798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fcd8c2376434248b9bbbd214127b9a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:58:56.767761 20798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fcd8c2376434248b9bbbd214127b9a8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:56.771137 20798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fcd8c2376434248b9bbbd214127b9a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fcd8c2376434248b9bbbd214127b9a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } }
I20260806 10:58:56.771914 20798 leader_election.cc:302] T 00000000000000000000000000000000 P 9fcd8c2376434248b9bbbd214127b9a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fcd8c2376434248b9bbbd214127b9a8; no voters:
I20260806 10:58:56.772671 20798 leader_election.cc:288] T 00000000000000000000000000000000 P 9fcd8c2376434248b9bbbd214127b9a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:58:56.772797 20808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fcd8c2376434248b9bbbd214127b9a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:58:56.775409 20798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fcd8c2376434248b9bbbd214127b9a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:58:56.779501 20808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fcd8c2376434248b9bbbd214127b9a8 [term 1 LEADER]: Becoming Leader. State: Replica: 9fcd8c2376434248b9bbbd214127b9a8, State: Running, Role: LEADER
I20260806 10:58:56.780316 20808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fcd8c2376434248b9bbbd214127b9a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fcd8c2376434248b9bbbd214127b9a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } }
I20260806 10:58:56.783905 20809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fcd8c2376434248b9bbbd214127b9a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fcd8c2376434248b9bbbd214127b9a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fcd8c2376434248b9bbbd214127b9a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } } }
I20260806 10:58:56.784404 20809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fcd8c2376434248b9bbbd214127b9a8 [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:56.784600 20810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fcd8c2376434248b9bbbd214127b9a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fcd8c2376434248b9bbbd214127b9a8. Latest consensus state: current_term: 1 leader_uuid: "9fcd8c2376434248b9bbbd214127b9a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fcd8c2376434248b9bbbd214127b9a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } } }
I20260806 10:58:56.784920 20810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fcd8c2376434248b9bbbd214127b9a8 [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:56.789268 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:56.770996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:54149 (local address 127.0.0.1:37879)
0806 10:58:56.771228 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:56.771235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:56.771857 (+ 622us) server_negotiation.cc:407] Connection header received
0806 10:58:56.772060 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:56.772064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:56.772148 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:56.772269 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:56.774135 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:56.775017 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:56.776379 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:56.776821 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:56.776937 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:56.779572 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:56.779577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:56.779581 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:58:56.779622 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:56.783284 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:56.785199 (+ 1915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:56.785204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:56.785207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:56.785410 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:58:56.786064 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:58:56.786070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:58:56.786073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:58:56.786614 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:56.788327 (+ 1713us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:56.788569 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 10:58:56.788830 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 10:58:56.795029 20815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:58:56.801021 20815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:58:56.811079 20815 catalog_manager.cc:1347] Generated new cluster ID: 46cf3325a12246d1871722e217d8d9c8
I20260806 10:58:56.811221 20815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:58:56.822196 20815 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:58:56.823668 20815 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:58:56.836890 20815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fcd8c2376434248b9bbbd214127b9a8: Generated new TSK 0
I20260806 10:58:56.837962 20815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:58:56.851936 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 20731
I20260806 10:58:56.853222 20793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:58:56.946619 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 20731
W20260806 10:58:56.965380 20716 connection.cc:535] client connection to 127.0.0.1:37879 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37879): Connection reset by peer (error 104)
2026-08-06T10:58:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2150 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:58:57 dist-test-slave-49h3 krb5kdc[20832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:58:57 dist-test-slave-49h3 krb5kdc[20832](info): set up 2 sockets
Aug 06 10:58:57 dist-test-slave-49h3 krb5kdc[20832](info): commencing operation
krb5kdc: starting...
W20260806 10:58:58.055506 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.002s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:58:58 dist-test-slave-49h3 krb5kdc[20832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:58:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:58:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 10:58:58.438917 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46473
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:58:58.834713 20846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:58:58.835095 20846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:58:58.835199 20846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:58:58.843089 20846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:58:58.843220 20846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:58:58.843279 20846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:58:58.843361 20846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:58:58.843410 20846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:58:58.854210 20846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:58:58.855830 20846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:58:58.857431 20846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:58:58.870270 20851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:58.870366 20854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:58:58.870411 20852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:58:58.871640 20846 server_base.cc:1034] running on GCE node
I20260806 10:58:58.872496 20846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:58:58.874085 20846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:58:58.875334 20846 hybrid_clock.cc:648] HybridClock initialized: now 1786013938875253 us; error 39 us; skew 500 ppm
I20260806 10:58:58.875730 20846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:58:58 dist-test-slave-49h3 krb5kdc[20832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:58:58.882639 20846 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 10:58:58.884366 20846 webserver.cc:460] Webserver started at http://127.0.0.1:33459/ using document root <none> and password file <none>
I20260806 10:58:58.885145 20846 fs_manager.cc:359] Metadata directory not provided
I20260806 10:58:58.885272 20846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:58:58.885659 20846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:58:58.889155 20846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "ccdd7eae4d3c40ae8ac026b267b7b62e"
format_stamp: "Formatted at 2026-08-06 10:58:58 on dist-test-slave-49h3"
server_key: "3dda46899350bb4883347e6de82fe5cd"
server_key_iv: "8ebab58b4b969bcbdc87826dcf96f294"
server_key_version: "encryptionkey@0"
I20260806 10:58:58.890074 20846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "ccdd7eae4d3c40ae8ac026b267b7b62e"
format_stamp: "Formatted at 2026-08-06 10:58:58 on dist-test-slave-49h3"
server_key: "3dda46899350bb4883347e6de82fe5cd"
server_key_iv: "8ebab58b4b969bcbdc87826dcf96f294"
server_key_version: "encryptionkey@0"
I20260806 10:58:58.896202 20846 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 10:58:58.900203 20861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:58:58.901857 20846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:58:58.902067 20846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "ccdd7eae4d3c40ae8ac026b267b7b62e"
format_stamp: "Formatted at 2026-08-06 10:58:58 on dist-test-slave-49h3"
server_key: "3dda46899350bb4883347e6de82fe5cd"
server_key_iv: "8ebab58b4b969bcbdc87826dcf96f294"
server_key_version: "encryptionkey@0"
I20260806 10:58:58.902282 20846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:58:58.914170 20846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:58:58.918715 20846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:58:58.919018 20846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:58:58.939814 20846 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46473
I20260806 10:58:58.939829 20913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46473 every 8 connection(s)
I20260806 10:58:58.941891 20846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:58:58.947237 20914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:58:58.950032 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 20846
I20260806 10:58:58.950264 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:58:58.950865 31592 external_mini_cluster.cc:1428] Setting key 17f06ca3b97a9162a91e5447c205cfe7
I20260806 10:58:58.961603 20914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccdd7eae4d3c40ae8ac026b267b7b62e: Bootstrap starting.
I20260806 10:58:58.965955 20914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccdd7eae4d3c40ae8ac026b267b7b62e: Neither blocks nor log segments found. Creating new log.
I20260806 10:58:58.967308 20914 log.cc:824] T 00000000000000000000000000000000 P ccdd7eae4d3c40ae8ac026b267b7b62e: Log is configured to *not* fsync() on all Append() calls
I20260806 10:58:58.967409 20917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:58.953359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:55067 (local address 127.0.0.1:46473)
0806 10:58:58.954445 (+ 1086us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:58.954462 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:58.954524 (+ 62us) server_negotiation.cc:407] Connection header received
0806 10:58:58.956620 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:58.956650 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:58.957198 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:58.957720 (+ 522us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:58.958463 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:58.960026 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:58.962044 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:58.962813 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:58.962969 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:58.964526 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:58.964535 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:58.964552 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:58.965232 (+ 680us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:58.965370 (+ 138us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:58.965403 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:58.965609 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:58.965875 (+ 266us) server_negotiation.cc:299] Negotiation successful
0806 10:58:58.966111 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":449,"threads_started":1}
I20260806 10:58:58.970820 20914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccdd7eae4d3c40ae8ac026b267b7b62e: No bootstrap required, opened a new log
I20260806 10:58:58.971932 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:58:58.979732 20914 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccdd7eae4d3c40ae8ac026b267b7b62e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccdd7eae4d3c40ae8ac026b267b7b62e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46473 } }
I20260806 10:58:58.980168 20914 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccdd7eae4d3c40ae8ac026b267b7b62e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:58:58.980266 20914 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccdd7eae4d3c40ae8ac026b267b7b62e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccdd7eae4d3c40ae8ac026b267b7b62e, State: Initialized, Role: FOLLOWER
I20260806 10:58:58.980928 20914 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccdd7eae4d3c40ae8ac026b267b7b62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccdd7eae4d3c40ae8ac026b267b7b62e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46473 } }
I20260806 10:58:58.981211 20914 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccdd7eae4d3c40ae8ac026b267b7b62e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:58:58.981364 20914 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccdd7eae4d3c40ae8ac026b267b7b62e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:58:58.981601 20914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccdd7eae4d3c40ae8ac026b267b7b62e [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:58:58.983533 20917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:58:58.976667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:39075 (local address 127.0.0.1:46473)
0806 10:58:58.976832 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 10:58:58.976837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:58:58.977419 (+ 582us) server_negotiation.cc:407] Connection header received
0806 10:58:58.977650 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:58:58.977655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:58:58.977732 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:58:58.977872 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:58:58.978430 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:58.979256 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:58.980764 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:58:58.981263 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:58:58.981380 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:58:58.982712 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:58:58.982717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:58:58.982721 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:58:58.982738 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:58:58.982810 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:58:58.982827 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:58:58.982914 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 10:58:58.983111 (+ 197us) server_negotiation.cc:299] Negotiation successful
0806 10:58:58.983161 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 10:58:58.984732 20914 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccdd7eae4d3c40ae8ac026b267b7b62e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccdd7eae4d3c40ae8ac026b267b7b62e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46473 } }
I20260806 10:58:58.985327 20914 leader_election.cc:302] T 00000000000000000000000000000000 P ccdd7eae4d3c40ae8ac026b267b7b62e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccdd7eae4d3c40ae8ac026b267b7b62e; no voters:
I20260806 10:58:58.985934 20914 leader_election.cc:288] T 00000000000000000000000000000000 P ccdd7eae4d3c40ae8ac026b267b7b62e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:58:58.986066 20924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccdd7eae4d3c40ae8ac026b267b7b62e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:58:58.986766 20924 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccdd7eae4d3c40ae8ac026b267b7b62e [term 1 LEADER]: Becoming Leader. State: Replica: ccdd7eae4d3c40ae8ac026b267b7b62e, State: Running, Role: LEADER
I20260806 10:58:58.987298 20924 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccdd7eae4d3c40ae8ac026b267b7b62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccdd7eae4d3c40ae8ac026b267b7b62e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46473 } }
I20260806 10:58:58.988225 20914 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccdd7eae4d3c40ae8ac026b267b7b62e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:58:58.990248 20925 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccdd7eae4d3c40ae8ac026b267b7b62e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccdd7eae4d3c40ae8ac026b267b7b62e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccdd7eae4d3c40ae8ac026b267b7b62e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46473 } } }
I20260806 10:58:58.990759 20925 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccdd7eae4d3c40ae8ac026b267b7b62e [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:58.990238 20926 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccdd7eae4d3c40ae8ac026b267b7b62e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccdd7eae4d3c40ae8ac026b267b7b62e. Latest consensus state: current_term: 1 leader_uuid: "ccdd7eae4d3c40ae8ac026b267b7b62e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccdd7eae4d3c40ae8ac026b267b7b62e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46473 } } }
I20260806 10:58:58.991120 20926 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccdd7eae4d3c40ae8ac026b267b7b62e [sys.catalog]: This master's current role is: LEADER
I20260806 10:58:58.993477 20929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:58:58.999351 20929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:58:59.009321 20929 catalog_manager.cc:1347] Generated new cluster ID: 48b21830ccf44ad08e1a0526c346335b
I20260806 10:58:59.009477 20929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:58:59.021191 20929 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:58:59.022837 20929 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:58:59.039639 20929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccdd7eae4d3c40ae8ac026b267b7b62e: Generated new TSK 0
I20260806 10:58:59.040807 20929 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:58:59.051184 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 20846
W20260806 10:58:59.052899 20909 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.148:55067
W20260806 10:58:59.053763 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 20846: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:58:59.053936 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 20846
W20260806 10:58:59.077862 20830 connection.cc:535] client connection to 127.0.0.1:46473 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46473): Connection reset by peer (error 104)
2026-08-06T10:58:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2112 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:58:59 dist-test-slave-49h3 krb5kdc[20946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:58:59 dist-test-slave-49h3 krb5kdc[20946](info): set up 2 sockets
Aug 06 10:58:59 dist-test-slave-49h3 krb5kdc[20946](info): commencing operation
krb5kdc: starting...
W20260806 10:59:00.175755 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.004s sys 0.064s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:59:00 dist-test-slave-49h3 krb5kdc[20946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:59:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:59:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 10:59:00.540627 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38399
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:59:00.929260 20959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:59:00.929690 20959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:59:00.929916 20959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:59:00.937779 20959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:59:00.937918 20959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:59:00.937966 20959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:59:00.938037 20959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:59:00.938083 20959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:59:00.948778 20959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:59:00.950263 20959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:59:00.951690 20959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:59:00.964545 20967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:00.964586 20965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:00.964476 20964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:59:00.965345 20959 server_base.cc:1034] running on GCE node
I20260806 10:59:00.966281 20959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:59:00.967943 20959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:59:00.969193 20959 hybrid_clock.cc:648] HybridClock initialized: now 1786013940969146 us; error 49 us; skew 500 ppm
I20260806 10:59:00.969686 20959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:59:00 dist-test-slave-49h3 krb5kdc[20946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:59:00.973146 20959 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 10:59:00.974972 20959 webserver.cc:460] Webserver started at http://127.0.0.1:44737/ using document root <none> and password file <none>
I20260806 10:59:00.975749 20959 fs_manager.cc:359] Metadata directory not provided
I20260806 10:59:00.975888 20959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:59:00.976220 20959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:59:00.979640 20959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "09b324b8496e4baa906f2ec2c5cd9fec"
format_stamp: "Formatted at 2026-08-06 10:59:00 on dist-test-slave-49h3"
server_key: "de9e9ad8cc6bb39ad9005ca72638759a"
server_key_iv: "c1f9895acafd59549408c64500574c3e"
server_key_version: "encryptionkey@0"
I20260806 10:59:00.980520 20959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "09b324b8496e4baa906f2ec2c5cd9fec"
format_stamp: "Formatted at 2026-08-06 10:59:00 on dist-test-slave-49h3"
server_key: "de9e9ad8cc6bb39ad9005ca72638759a"
server_key_iv: "c1f9895acafd59549408c64500574c3e"
server_key_version: "encryptionkey@0"
I20260806 10:59:00.986886 20959 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260806 10:59:00.991015 20975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:59:00.992664 20959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:59:00.992877 20959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "09b324b8496e4baa906f2ec2c5cd9fec"
format_stamp: "Formatted at 2026-08-06 10:59:00 on dist-test-slave-49h3"
server_key: "de9e9ad8cc6bb39ad9005ca72638759a"
server_key_iv: "c1f9895acafd59549408c64500574c3e"
server_key_version: "encryptionkey@0"
I20260806 10:59:00.993109 20959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:59:00.999842 20959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:59:01.004668 20959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:59:01.005000 20959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:59:01.024212 20959 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38399
I20260806 10:59:01.024235 21028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38399 every 8 connection(s)
I20260806 10:59:01.026292 20959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:59:01.028963 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 20959
I20260806 10:59:01.029161 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:59:01.029814 31592 external_mini_cluster.cc:1428] Setting key f4b4b0f2e64199b0f32a768d0c125fb0
I20260806 10:59:01.032676 21029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:59:01.038533 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:01.032008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:57589 (local address 127.0.0.1:38399)
0806 10:59:01.032893 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:01.032907 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:01.032956 (+ 49us) server_negotiation.cc:407] Connection header received
0806 10:59:01.034735 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:01.034762 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:01.035163 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:01.035553 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:01.036751 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:01.036757 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:01.036774 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:59:01.036790 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:01.036879 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:59:01.036900 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:01.037077 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:01.037208 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 10:59:01.037421 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":373,"threads_started":1}
I20260806 10:59:01.042550 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:59:01.045945 21029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09b324b8496e4baa906f2ec2c5cd9fec: Bootstrap starting.
I20260806 10:59:01.050568 21029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09b324b8496e4baa906f2ec2c5cd9fec: Neither blocks nor log segments found. Creating new log.
I20260806 10:59:01.051481 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:01.048730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:35595 (local address 127.0.0.1:38399)
0806 10:59:01.048943 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:01.048949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:01.049292 (+ 343us) server_negotiation.cc:407] Connection header received
0806 10:59:01.049540 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:01.049543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:01.049601 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:01.049685 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:01.050778 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:01.050782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:01.050786 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:59:01.050807 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:01.050852 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:59:01.050867 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:01.050967 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:01.051122 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 10:59:01.051196 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 10:59:01.052073 21029 log.cc:824] T 00000000000000000000000000000000 P 09b324b8496e4baa906f2ec2c5cd9fec: Log is configured to *not* fsync() on all Append() calls
I20260806 10:59:01.055307 21029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09b324b8496e4baa906f2ec2c5cd9fec: No bootstrap required, opened a new log
I20260806 10:59:01.063632 21029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09b324b8496e4baa906f2ec2c5cd9fec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b324b8496e4baa906f2ec2c5cd9fec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38399 } }
I20260806 10:59:01.064067 21029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09b324b8496e4baa906f2ec2c5cd9fec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:59:01.064177 21029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09b324b8496e4baa906f2ec2c5cd9fec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09b324b8496e4baa906f2ec2c5cd9fec, State: Initialized, Role: FOLLOWER
I20260806 10:59:01.064805 21029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09b324b8496e4baa906f2ec2c5cd9fec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b324b8496e4baa906f2ec2c5cd9fec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38399 } }
I20260806 10:59:01.065039 21029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09b324b8496e4baa906f2ec2c5cd9fec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:59:01.065182 21029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09b324b8496e4baa906f2ec2c5cd9fec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:59:01.065356 21029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09b324b8496e4baa906f2ec2c5cd9fec [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:59:01.068035 21029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09b324b8496e4baa906f2ec2c5cd9fec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b324b8496e4baa906f2ec2c5cd9fec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38399 } }
I20260806 10:59:01.068648 21029 leader_election.cc:302] T 00000000000000000000000000000000 P 09b324b8496e4baa906f2ec2c5cd9fec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09b324b8496e4baa906f2ec2c5cd9fec; no voters:
I20260806 10:59:01.069314 21029 leader_election.cc:288] T 00000000000000000000000000000000 P 09b324b8496e4baa906f2ec2c5cd9fec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:59:01.069566 21039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09b324b8496e4baa906f2ec2c5cd9fec [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:59:01.070201 21039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09b324b8496e4baa906f2ec2c5cd9fec [term 1 LEADER]: Becoming Leader. State: Replica: 09b324b8496e4baa906f2ec2c5cd9fec, State: Running, Role: LEADER
I20260806 10:59:01.070801 21039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09b324b8496e4baa906f2ec2c5cd9fec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b324b8496e4baa906f2ec2c5cd9fec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38399 } }
I20260806 10:59:01.071586 21029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09b324b8496e4baa906f2ec2c5cd9fec [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:59:01.074025 21041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09b324b8496e4baa906f2ec2c5cd9fec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09b324b8496e4baa906f2ec2c5cd9fec. Latest consensus state: current_term: 1 leader_uuid: "09b324b8496e4baa906f2ec2c5cd9fec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b324b8496e4baa906f2ec2c5cd9fec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38399 } } }
I20260806 10:59:01.074494 21041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09b324b8496e4baa906f2ec2c5cd9fec [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:01.074497 21040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09b324b8496e4baa906f2ec2c5cd9fec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09b324b8496e4baa906f2ec2c5cd9fec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b324b8496e4baa906f2ec2c5cd9fec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38399 } } }
I20260806 10:59:01.076016 21040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09b324b8496e4baa906f2ec2c5cd9fec [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:01.077328 21048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:59:01.083374 21048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:59:01.094435 21048 catalog_manager.cc:1347] Generated new cluster ID: ed7f60eff0cb498289001bbc037b9d5b
I20260806 10:59:01.094597 21048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:59:01.119302 21048 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:59:01.119817 21048 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:59:01.128726 21048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09b324b8496e4baa906f2ec2c5cd9fec: Generated new TSK 0
I20260806 10:59:01.129793 21048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:59:01.150386 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 20959
W20260806 10:59:01.151877 21024 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.148:57589
W20260806 10:59:01.152851 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 20959: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:59:01.153019 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 20959
2026-08-06T10:59:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2094 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:59:01 dist-test-slave-49h3 krb5kdc[21062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:59:01 dist-test-slave-49h3 krb5kdc[21062](info): set up 2 sockets
Aug 06 10:59:01 dist-test-slave-49h3 krb5kdc[21062](info): commencing operation
krb5kdc: starting...
W20260806 10:59:02.249218 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.004s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:59:02 dist-test-slave-49h3 krb5kdc[21062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:59:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:59:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 10:59:02.644896 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44921
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:59:03.044833 21076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:59:03.045250 21076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:59:03.045410 21076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:59:03.053577 21076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:59:03.053740 21076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:59:03.053826 21076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:59:03.053918 21076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:59:03.053987 21076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:59:03.065173 21076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44921
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:59:03.067026 21076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:59:03.068647 21076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:59:03.081975 21081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:03.081925 21084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:03.082028 21082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:59:03.083144 21076 server_base.cc:1034] running on GCE node
I20260806 10:59:03.084038 21076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:59:03.085843 21076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:59:03.087052 21076 hybrid_clock.cc:648] HybridClock initialized: now 1786013943087005 us; error 36 us; skew 500 ppm
I20260806 10:59:03.087450 21076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:59:03 dist-test-slave-49h3 krb5kdc[21062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:59:03.091354 21076 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 10:59:03.093297 21076 webserver.cc:460] Webserver started at http://127.0.0.1:43811/ using document root <none> and password file <none>
I20260806 10:59:03.094214 21076 fs_manager.cc:359] Metadata directory not provided
I20260806 10:59:03.094558 21076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:59:03.094981 21076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:59:03.098851 21076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "7e54759a321043ed8586c879a887aaec"
format_stamp: "Formatted at 2026-08-06 10:59:03 on dist-test-slave-49h3"
server_key: "5f85e6ab54448475cdbfb61152dff1b3"
server_key_iv: "163aed5ab02aa0bc8a2d28a4b778ce99"
server_key_version: "encryptionkey@0"
I20260806 10:59:03.099766 21076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "7e54759a321043ed8586c879a887aaec"
format_stamp: "Formatted at 2026-08-06 10:59:03 on dist-test-slave-49h3"
server_key: "5f85e6ab54448475cdbfb61152dff1b3"
server_key_iv: "163aed5ab02aa0bc8a2d28a4b778ce99"
server_key_version: "encryptionkey@0"
I20260806 10:59:03.106477 21076 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 10:59:03.110733 21091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:59:03.112722 21076 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 10:59:03.113010 21076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "7e54759a321043ed8586c879a887aaec"
format_stamp: "Formatted at 2026-08-06 10:59:03 on dist-test-slave-49h3"
server_key: "5f85e6ab54448475cdbfb61152dff1b3"
server_key_iv: "163aed5ab02aa0bc8a2d28a4b778ce99"
server_key_version: "encryptionkey@0"
I20260806 10:59:03.113276 21076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:59:03.144785 21076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:59:03.149761 21076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:59:03.150085 21076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:59:03.170714 21076 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44921
I20260806 10:59:03.170722 21144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44921 every 8 connection(s)
I20260806 10:59:03.172514 21076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:59:03.176668 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 21076
I20260806 10:59:03.176887 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:59:03.177138 21145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:59:03.177479 31592 external_mini_cluster.cc:1428] Setting key 75afcc817e6eae5fe7959c3b78f5db99
I20260806 10:59:03.193563 21145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e54759a321043ed8586c879a887aaec: Bootstrap starting.
Aug 06 10:59:03 dist-test-slave-49h3 krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 10:59:03.197860 21145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e54759a321043ed8586c879a887aaec: Neither blocks nor log segments found. Creating new log.
I20260806 10:59:03.199237 21145 log.cc:824] T 00000000000000000000000000000000 P 7e54759a321043ed8586c879a887aaec: Log is configured to *not* fsync() on all Append() calls
I20260806 10:59:03.203542 21145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e54759a321043ed8586c879a887aaec: No bootstrap required, opened a new log
I20260806 10:59:03.204484 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:03.179696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:42961 (local address 127.0.0.1:44921)
0806 10:59:03.180633 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:03.180646 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:03.180693 (+ 47us) server_negotiation.cc:407] Connection header received
0806 10:59:03.182465 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:03.182490 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:03.182967 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:03.183357 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:03.185378 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:03.186892 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:03.188753 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:03.189614 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:03.189771 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:03.195396 (+ 5625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:03.195405 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:03.195422 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:03.195473 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:03.198896 (+ 3423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:03.200132 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:03.200146 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:03.200157 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:03.200479 (+ 322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:03.200969 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:03.200976 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:03.200979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:03.201743 (+ 764us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:03.202039 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:03.202659 (+ 620us) server_negotiation.cc:299] Negotiation successful
0806 10:59:03.203049 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":388,"threads_started":1}
I20260806 10:59:03.209685 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:59:03.213088 21145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e54759a321043ed8586c879a887aaec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e54759a321043ed8586c879a887aaec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44921 } }
I20260806 10:59:03.213692 21145 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e54759a321043ed8586c879a887aaec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:59:03.213819 21145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e54759a321043ed8586c879a887aaec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e54759a321043ed8586c879a887aaec, State: Initialized, Role: FOLLOWER
I20260806 10:59:03.214540 21145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e54759a321043ed8586c879a887aaec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e54759a321043ed8586c879a887aaec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44921 } }
I20260806 10:59:03.214814 21145 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e54759a321043ed8586c879a887aaec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:59:03.214993 21145 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e54759a321043ed8586c879a887aaec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:59:03.215207 21145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e54759a321043ed8586c879a887aaec [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:59:03.218251 21145 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e54759a321043ed8586c879a887aaec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e54759a321043ed8586c879a887aaec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44921 } }
I20260806 10:59:03.219002 21145 leader_election.cc:302] T 00000000000000000000000000000000 P 7e54759a321043ed8586c879a887aaec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e54759a321043ed8586c879a887aaec; no voters:
I20260806 10:59:03.219802 21145 leader_election.cc:288] T 00000000000000000000000000000000 P 7e54759a321043ed8586c879a887aaec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:59:03.219916 21155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e54759a321043ed8586c879a887aaec [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:59:03.221218 21155 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e54759a321043ed8586c879a887aaec [term 1 LEADER]: Becoming Leader. State: Replica: 7e54759a321043ed8586c879a887aaec, State: Running, Role: LEADER
I20260806 10:59:03.221992 21155 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e54759a321043ed8586c879a887aaec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e54759a321043ed8586c879a887aaec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44921 } }
I20260806 10:59:03.222819 21145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e54759a321043ed8586c879a887aaec [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:59:03.225456 21156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e54759a321043ed8586c879a887aaec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e54759a321043ed8586c879a887aaec. Latest consensus state: current_term: 1 leader_uuid: "7e54759a321043ed8586c879a887aaec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e54759a321043ed8586c879a887aaec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44921 } } }
I20260806 10:59:03.225916 21156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e54759a321043ed8586c879a887aaec [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:03.227679 21157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e54759a321043ed8586c879a887aaec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e54759a321043ed8586c879a887aaec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e54759a321043ed8586c879a887aaec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44921 } } }
I20260806 10:59:03.228056 21157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e54759a321043ed8586c879a887aaec [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:03.231731 21160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:59:03.232913 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:03.214936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:60277 (local address 127.0.0.1:44921)
0806 10:59:03.215158 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:03.215169 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:03.215718 (+ 549us) server_negotiation.cc:407] Connection header received
0806 10:59:03.215893 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:03.215898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:03.216005 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:03.216130 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:03.217804 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:03.219044 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:03.220705 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:03.221469 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:03.221594 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:03.224665 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:03.224673 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:03.224677 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:03.224722 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:03.228705 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:03.229895 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:03.229901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:03.229905 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:03.230073 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:03.230731 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:03.230736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:03.230739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:03.231255 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:03.231593 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:03.232052 (+ 459us) server_negotiation.cc:299] Negotiation successful
0806 10:59:03.232247 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 10:59:03.238507 21160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:59:03.248485 21160 catalog_manager.cc:1347] Generated new cluster ID: af1f209c4d814f45911a0d9236c322c2
I20260806 10:59:03.248641 21160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:59:03.284647 21160 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:59:03.286938 21160 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:59:03.307072 21160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e54759a321043ed8586c879a887aaec: Generated new TSK 0
I20260806 10:59:03.308326 21160 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:59:03.322526 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 21076
I20260806 10:59:03.324004 21140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:59:03.430073 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 21076
W20260806 10:59:03.450258 21060 connection.cc:535] client connection to 127.0.0.1:44921 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44921): Connection reset by peer (error 104)
2026-08-06T10:59:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2282 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:59:03 dist-test-slave-49h3 krb5kdc[21179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:59:03 dist-test-slave-49h3 krb5kdc[21179](info): set up 2 sockets
Aug 06 10:59:03 dist-test-slave-49h3 krb5kdc[21179](info): commencing operation
krb5kdc: starting...
W20260806 10:59:04.555687 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:59:04 dist-test-slave-49h3 krb5kdc[21179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:59:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:59:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 10:59:04.920540 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33595
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:59:05.305425 21193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:59:05.305838 21193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:59:05.305982 21193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:59:05.313753 21193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:59:05.313922 21193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:59:05.314009 21193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:59:05.314098 21193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:59:05.314164 21193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:59:05.324829 21193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33595
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:59:05.326432 21193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:59:05.327894 21193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:59:05.340663 21198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:05.340615 21199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:05.340593 21201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:59:05.341665 21193 server_base.cc:1034] running on GCE node
I20260806 10:59:05.342582 21193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:59:05.344363 21193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:59:05.345577 21193 hybrid_clock.cc:648] HybridClock initialized: now 1786013945345535 us; error 31 us; skew 500 ppm
I20260806 10:59:05.345973 21193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:59:05 dist-test-slave-49h3 krb5kdc[21179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:59:05.349579 21193 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 10:59:05.351317 21193 webserver.cc:460] Webserver started at http://127.0.0.1:37115/ using document root <none> and password file <none>
I20260806 10:59:05.352082 21193 fs_manager.cc:359] Metadata directory not provided
I20260806 10:59:05.352226 21193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:59:05.352571 21193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:59:05.356045 21193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "012a799b45204363930dbd7c35367917"
format_stamp: "Formatted at 2026-08-06 10:59:05 on dist-test-slave-49h3"
server_key: "21726b2bb896b513a72524450844333f"
server_key_iv: "7fbfd53e20a7559017db704c38c795f9"
server_key_version: "encryptionkey@0"
I20260806 10:59:05.356998 21193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "012a799b45204363930dbd7c35367917"
format_stamp: "Formatted at 2026-08-06 10:59:05 on dist-test-slave-49h3"
server_key: "21726b2bb896b513a72524450844333f"
server_key_iv: "7fbfd53e20a7559017db704c38c795f9"
server_key_version: "encryptionkey@0"
I20260806 10:59:05.363258 21193 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260806 10:59:05.367391 21209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:59:05.369081 21193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 10:59:05.369297 21193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "012a799b45204363930dbd7c35367917"
format_stamp: "Formatted at 2026-08-06 10:59:05 on dist-test-slave-49h3"
server_key: "21726b2bb896b513a72524450844333f"
server_key_iv: "7fbfd53e20a7559017db704c38c795f9"
server_key_version: "encryptionkey@0"
I20260806 10:59:05.369603 21193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:59:05.396404 21193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:59:05.401369 21193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:59:05.401746 21193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:59:05.420615 21193 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33595
I20260806 10:59:05.420639 21260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33595 every 8 connection(s)
I20260806 10:59:05.422231 21193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:59:05.426678 21261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:59:05.428574 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 21193
I20260806 10:59:05.428758 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:59:05.429289 31592 external_mini_cluster.cc:1428] Setting key 0b58410192bc9f398d0f0e6f226e1915
I20260806 10:59:05.442734 21261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 012a799b45204363930dbd7c35367917: Bootstrap starting.
I20260806 10:59:05.445974 21264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:05.431427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:47363 (local address 127.0.0.1:33595)
0806 10:59:05.432349 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:05.432365 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:05.432433 (+ 68us) server_negotiation.cc:407] Connection header received
0806 10:59:05.434385 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:05.434424 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:05.435141 (+ 717us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:05.435649 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:05.436279 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:05.438343 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:05.440027 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:05.441262 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:05.441505 (+ 243us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:05.442828 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:05.442841 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:05.442864 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:59:05.443666 (+ 802us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:05.443789 (+ 123us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:59:05.443826 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:05.443998 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:05.444277 (+ 279us) server_negotiation.cc:299] Negotiation successful
0806 10:59:05.444500 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":359,"threads_started":1}
I20260806 10:59:05.447137 21261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 012a799b45204363930dbd7c35367917: Neither blocks nor log segments found. Creating new log.
I20260806 10:59:05.448769 21261 log.cc:824] T 00000000000000000000000000000000 P 012a799b45204363930dbd7c35367917: Log is configured to *not* fsync() on all Append() calls
I20260806 10:59:05.451043 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:59:05.452453 21261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 012a799b45204363930dbd7c35367917: No bootstrap required, opened a new log
I20260806 10:59:05.462941 21261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 012a799b45204363930dbd7c35367917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012a799b45204363930dbd7c35367917" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33595 } }
I20260806 10:59:05.463668 21261 raft_consensus.cc:374] T 00000000000000000000000000000000 P 012a799b45204363930dbd7c35367917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:59:05.463748 21264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:05.456806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:53717 (local address 127.0.0.1:33595)
0806 10:59:05.457056 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:05.457063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:05.457283 (+ 220us) server_negotiation.cc:407] Connection header received
0806 10:59:05.457441 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:05.457468 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:05.457556 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:05.457681 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:05.458198 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:05.459381 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:05.460822 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:05.461609 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:05.461758 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:05.462957 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:05.462963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:05.462968 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:59:05.462994 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:05.463078 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:59:05.463101 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:05.463224 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:05.463360 (+ 136us) server_negotiation.cc:299] Negotiation successful
0806 10:59:05.463431 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 10:59:05.463811 21261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 012a799b45204363930dbd7c35367917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012a799b45204363930dbd7c35367917, State: Initialized, Role: FOLLOWER
I20260806 10:59:05.464618 21261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 012a799b45204363930dbd7c35367917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012a799b45204363930dbd7c35367917" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33595 } }
I20260806 10:59:05.464907 21261 raft_consensus.cc:388] T 00000000000000000000000000000000 P 012a799b45204363930dbd7c35367917 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:59:05.465068 21261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 012a799b45204363930dbd7c35367917 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:59:05.465269 21261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 012a799b45204363930dbd7c35367917 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:59:05.467768 21261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 012a799b45204363930dbd7c35367917 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012a799b45204363930dbd7c35367917" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33595 } }
I20260806 10:59:05.468408 21261 leader_election.cc:302] T 00000000000000000000000000000000 P 012a799b45204363930dbd7c35367917 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 012a799b45204363930dbd7c35367917; no voters:
I20260806 10:59:05.469059 21261 leader_election.cc:288] T 00000000000000000000000000000000 P 012a799b45204363930dbd7c35367917 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:59:05.469228 21271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 012a799b45204363930dbd7c35367917 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:59:05.470510 21271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 012a799b45204363930dbd7c35367917 [term 1 LEADER]: Becoming Leader. State: Replica: 012a799b45204363930dbd7c35367917, State: Running, Role: LEADER
I20260806 10:59:05.471112 21271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 012a799b45204363930dbd7c35367917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012a799b45204363930dbd7c35367917" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33595 } }
I20260806 10:59:05.471616 21261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 012a799b45204363930dbd7c35367917 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:59:05.474952 21272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 012a799b45204363930dbd7c35367917 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "012a799b45204363930dbd7c35367917" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012a799b45204363930dbd7c35367917" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33595 } } }
I20260806 10:59:05.475396 21272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 012a799b45204363930dbd7c35367917 [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:05.477705 21273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 012a799b45204363930dbd7c35367917 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 012a799b45204363930dbd7c35367917. Latest consensus state: current_term: 1 leader_uuid: "012a799b45204363930dbd7c35367917" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012a799b45204363930dbd7c35367917" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33595 } } }
I20260806 10:59:05.478219 21273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 012a799b45204363930dbd7c35367917 [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:05.479815 21280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:59:05.486110 21280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:59:05.497833 21280 catalog_manager.cc:1347] Generated new cluster ID: df7bbd29b9eb4c0ab1950319408ca60e
I20260806 10:59:05.497988 21280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:59:05.526124 21280 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:59:05.528355 21280 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:59:05.542959 21280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 012a799b45204363930dbd7c35367917: Generated new TSK 0
I20260806 10:59:05.544235 21280 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:59:05.551717 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 21193
W20260806 10:59:05.553568 21256 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.148:47363
W20260806 10:59:05.554469 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 21193: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:59:05.554615 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 21193
W20260806 10:59:05.573383 21177 connection.cc:535] client connection to 127.0.0.1:33595 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33595): Connection reset by peer (error 104)
2026-08-06T10:59:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2118 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:59:05 dist-test-slave-49h3 krb5kdc[21292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:59:05 dist-test-slave-49h3 krb5kdc[21292](info): set up 2 sockets
Aug 06 10:59:05 dist-test-slave-49h3 krb5kdc[21292](info): commencing operation
krb5kdc: starting...
W20260806 10:59:06.634406 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:59:06 dist-test-slave-49h3 krb5kdc[21292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:59:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:59:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 10:59:06.995286 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:59:07.355221 21306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:59:07.355566 21306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:59:07.355698 21306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:59:07.363440 21306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:59:07.363581 21306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:59:07.363659 21306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:59:07.363747 21306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:59:07.363806 21306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:59:07.374552 21306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44201
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:59:07.376003 21306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:59:07.377480 21306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:59:07.389919 21315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:07.389961 21312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:07.389887 21313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:59:07.391010 21306 server_base.cc:1034] running on GCE node
I20260806 10:59:07.391896 21306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:59:07.393525 21306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:59:07.394759 21306 hybrid_clock.cc:648] HybridClock initialized: now 1786013947394712 us; error 46 us; skew 500 ppm
I20260806 10:59:07.395136 21306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:59:07 dist-test-slave-49h3 krb5kdc[21292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:59:07.398471 21306 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 10:59:07.400132 21306 webserver.cc:460] Webserver started at http://127.0.0.1:38935/ using document root <none> and password file <none>
I20260806 10:59:07.400986 21306 fs_manager.cc:359] Metadata directory not provided
I20260806 10:59:07.401129 21306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:59:07.401496 21306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:59:07.404915 21306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "e27b9826225b4c3cbac9f5557dfce797"
format_stamp: "Formatted at 2026-08-06 10:59:07 on dist-test-slave-49h3"
server_key: "2bc09e4a0d765e8dd545ace0dffbd632"
server_key_iv: "07515f8e13e00fde485f353b70873d74"
server_key_version: "encryptionkey@0"
I20260806 10:59:07.405877 21306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "e27b9826225b4c3cbac9f5557dfce797"
format_stamp: "Formatted at 2026-08-06 10:59:07 on dist-test-slave-49h3"
server_key: "2bc09e4a0d765e8dd545ace0dffbd632"
server_key_iv: "07515f8e13e00fde485f353b70873d74"
server_key_version: "encryptionkey@0"
I20260806 10:59:07.411944 21306 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 10:59:07.415936 21322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:59:07.417579 21306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:59:07.417779 21306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "e27b9826225b4c3cbac9f5557dfce797"
format_stamp: "Formatted at 2026-08-06 10:59:07 on dist-test-slave-49h3"
server_key: "2bc09e4a0d765e8dd545ace0dffbd632"
server_key_iv: "07515f8e13e00fde485f353b70873d74"
server_key_version: "encryptionkey@0"
I20260806 10:59:07.417997 21306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:59:07.424422 21306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:59:07.429199 21306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:59:07.429575 21306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:59:07.447234 21306 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44201
I20260806 10:59:07.447290 21375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44201 every 8 connection(s)
I20260806 10:59:07.448716 21306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:59:07.452140 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 21306
I20260806 10:59:07.452327 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:59:07.452739 21376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:59:07.452787 31592 external_mini_cluster.cc:1428] Setting key 01eab460275c74a7ff6f86caf5d1fc18
I20260806 10:59:07.462162 21379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:07.454710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:51587 (local address 127.0.0.1:44201)
0806 10:59:07.455557 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:07.455572 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:07.455640 (+ 68us) server_negotiation.cc:407] Connection header received
0806 10:59:07.457903 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:07.457936 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:07.458439 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:07.458957 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:07.460024 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:07.460032 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:07.460048 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:59:07.460067 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:07.460157 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:59:07.460187 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:07.460430 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:07.460572 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 10:59:07.460807 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":317,"threads_started":1}
I20260806 10:59:07.466084 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:59:07.467797 21376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e27b9826225b4c3cbac9f5557dfce797: Bootstrap starting.
I20260806 10:59:07.471369 21376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e27b9826225b4c3cbac9f5557dfce797: Neither blocks nor log segments found. Creating new log.
I20260806 10:59:07.472625 21376 log.cc:824] T 00000000000000000000000000000000 P e27b9826225b4c3cbac9f5557dfce797: Log is configured to *not* fsync() on all Append() calls
I20260806 10:59:07.472846 21379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:07.470480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:37303 (local address 127.0.0.1:44201)
0806 10:59:07.470696 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:07.470702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:07.470973 (+ 271us) server_negotiation.cc:407] Connection header received
0806 10:59:07.471118 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:07.471123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:07.471188 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:07.471332 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:07.472172 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:07.472177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:07.472182 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:59:07.472201 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:07.472250 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:59:07.472279 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:07.472405 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:07.472456 (+ 51us) server_negotiation.cc:299] Negotiation successful
0806 10:59:07.472515 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260806 10:59:07.475880 21376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e27b9826225b4c3cbac9f5557dfce797: No bootstrap required, opened a new log
I20260806 10:59:07.483738 21376 raft_consensus.cc:348] T 00000000000000000000000000000000 P e27b9826225b4c3cbac9f5557dfce797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e27b9826225b4c3cbac9f5557dfce797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44201 } }
I20260806 10:59:07.484171 21376 raft_consensus.cc:374] T 00000000000000000000000000000000 P e27b9826225b4c3cbac9f5557dfce797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:59:07.484268 21376 raft_consensus.cc:729] T 00000000000000000000000000000000 P e27b9826225b4c3cbac9f5557dfce797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e27b9826225b4c3cbac9f5557dfce797, State: Initialized, Role: FOLLOWER
I20260806 10:59:07.484905 21376 consensus_queue.cc:260] T 00000000000000000000000000000000 P e27b9826225b4c3cbac9f5557dfce797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e27b9826225b4c3cbac9f5557dfce797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44201 } }
I20260806 10:59:07.485153 21376 raft_consensus.cc:388] T 00000000000000000000000000000000 P e27b9826225b4c3cbac9f5557dfce797 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:59:07.485284 21376 raft_consensus.cc:482] T 00000000000000000000000000000000 P e27b9826225b4c3cbac9f5557dfce797 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:59:07.485432 21376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e27b9826225b4c3cbac9f5557dfce797 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:59:07.487742 21376 raft_consensus.cc:504] T 00000000000000000000000000000000 P e27b9826225b4c3cbac9f5557dfce797 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e27b9826225b4c3cbac9f5557dfce797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44201 } }
I20260806 10:59:07.488351 21376 leader_election.cc:302] T 00000000000000000000000000000000 P e27b9826225b4c3cbac9f5557dfce797 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e27b9826225b4c3cbac9f5557dfce797; no voters:
I20260806 10:59:07.489133 21376 leader_election.cc:288] T 00000000000000000000000000000000 P e27b9826225b4c3cbac9f5557dfce797 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:59:07.489333 21386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e27b9826225b4c3cbac9f5557dfce797 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:59:07.490051 21386 raft_consensus.cc:686] T 00000000000000000000000000000000 P e27b9826225b4c3cbac9f5557dfce797 [term 1 LEADER]: Becoming Leader. State: Replica: e27b9826225b4c3cbac9f5557dfce797, State: Running, Role: LEADER
I20260806 10:59:07.490603 21386 consensus_queue.cc:237] T 00000000000000000000000000000000 P e27b9826225b4c3cbac9f5557dfce797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e27b9826225b4c3cbac9f5557dfce797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44201 } }
I20260806 10:59:07.491312 21376 sys_catalog.cc:564] T 00000000000000000000000000000000 P e27b9826225b4c3cbac9f5557dfce797 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:59:07.493580 21387 sys_catalog.cc:455] T 00000000000000000000000000000000 P e27b9826225b4c3cbac9f5557dfce797 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e27b9826225b4c3cbac9f5557dfce797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e27b9826225b4c3cbac9f5557dfce797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44201 } } }
I20260806 10:59:07.494024 21387 sys_catalog.cc:458] T 00000000000000000000000000000000 P e27b9826225b4c3cbac9f5557dfce797 [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:07.498618 21388 sys_catalog.cc:455] T 00000000000000000000000000000000 P e27b9826225b4c3cbac9f5557dfce797 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e27b9826225b4c3cbac9f5557dfce797. Latest consensus state: current_term: 1 leader_uuid: "e27b9826225b4c3cbac9f5557dfce797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e27b9826225b4c3cbac9f5557dfce797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44201 } } }
I20260806 10:59:07.498973 21388 sys_catalog.cc:458] T 00000000000000000000000000000000 P e27b9826225b4c3cbac9f5557dfce797 [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:07.502650 21392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:59:07.507746 21392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:59:07.517513 21392 catalog_manager.cc:1347] Generated new cluster ID: f693110d5e0c4b359e0e5cfcc2b8c2a7
I20260806 10:59:07.517657 21392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:59:07.550150 21392 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:59:07.550823 21392 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:59:07.561806 21392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e27b9826225b4c3cbac9f5557dfce797: Generated new TSK 0
I20260806 10:59:07.563117 21392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:59:07.584111 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 21306
W20260806 10:59:07.585356 21371 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.148:51587
W20260806 10:59:07.586144 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 21306: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:59:07.586319 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 21306
2026-08-06T10:59:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2029 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:59:07 dist-test-slave-49h3 krb5kdc[21407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:59:07 dist-test-slave-49h3 krb5kdc[21407](info): set up 2 sockets
Aug 06 10:59:07 dist-test-slave-49h3 krb5kdc[21407](info): commencing operation
krb5kdc: starting...
W20260806 10:59:08.674762 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.007s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:59:08 dist-test-slave-49h3 krb5kdc[21407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:59:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:59:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:59:09.038795 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:35965
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:43153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:35965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:59:09.409179 21420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:59:09.409581 21420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:59:09.409737 21420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:59:09.417613 21420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:59:09.417774 21420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:59:09.417858 21420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:59:09.417945 21420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:59:09.418008 21420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:59:09.428747 21420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:43153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:35965
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:35965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:59:09.430243 21420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:59:09.431679 21420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:59:09.444515 21425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:09.444502 21426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:09.444592 21428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:59:09.444983 21420 server_base.cc:1034] running on GCE node
I20260806 10:59:09.446141 21420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:59:09.447707 21420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:59:09.448916 21420 hybrid_clock.cc:648] HybridClock initialized: now 1786013949448846 us; error 53 us; skew 500 ppm
I20260806 10:59:09.449381 21420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:59:09 dist-test-slave-49h3 krb5kdc[21407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013949, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:59:09.458446 21420 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:59:09.460153 21420 webserver.cc:460] Webserver started at http://127.30.218.62:43667/ using document root <none> and password file <none>
I20260806 10:59:09.460897 21420 fs_manager.cc:359] Metadata directory not provided
I20260806 10:59:09.461009 21420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:59:09.461298 21420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:59:09.464659 21420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "353baacb04dd4a1eb67307ee9ad06f7a"
format_stamp: "Formatted at 2026-08-06 10:59:09 on dist-test-slave-49h3"
server_key: "c32c18daebc9ae20f935a3b7f6c6592d"
server_key_iv: "261680884441b4d6adecf790b3d466c5"
server_key_version: "encryptionkey@0"
I20260806 10:59:09.465559 21420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "353baacb04dd4a1eb67307ee9ad06f7a"
format_stamp: "Formatted at 2026-08-06 10:59:09 on dist-test-slave-49h3"
server_key: "c32c18daebc9ae20f935a3b7f6c6592d"
server_key_iv: "261680884441b4d6adecf790b3d466c5"
server_key_version: "encryptionkey@0"
I20260806 10:59:09.471388 21420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 10:59:09.475510 21435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:59:09.477035 21420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 10:59:09.477240 21420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "353baacb04dd4a1eb67307ee9ad06f7a"
format_stamp: "Formatted at 2026-08-06 10:59:09 on dist-test-slave-49h3"
server_key: "c32c18daebc9ae20f935a3b7f6c6592d"
server_key_iv: "261680884441b4d6adecf790b3d466c5"
server_key_version: "encryptionkey@0"
I20260806 10:59:09.477471 21420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:59:09.501969 21420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:59:09.506445 21420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:59:09.506724 21420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:59:09.524693 21420 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:35965
I20260806 10:59:09.524715 21490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:35965 every 8 connection(s)
I20260806 10:59:09.526232 21420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:59:09.528041 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 21420
I20260806 10:59:09.528252 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:59:09.528695 31592 external_mini_cluster.cc:1428] Setting key e90632f0c1e3840ad31f899ddcec7307
I20260806 10:59:09.530706 21491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 10:59:09 dist-test-slave-49h3 krb5kdc[21407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:59:09.545065 21491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353baacb04dd4a1eb67307ee9ad06f7a: Bootstrap starting.
I20260806 10:59:09.549616 21491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 353baacb04dd4a1eb67307ee9ad06f7a: Neither blocks nor log segments found. Creating new log.
I20260806 10:59:09.551059 21491 log.cc:824] T 00000000000000000000000000000000 P 353baacb04dd4a1eb67307ee9ad06f7a: Log is configured to *not* fsync() on all Append() calls
I20260806 10:59:09.555078 21491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353baacb04dd4a1eb67307ee9ad06f7a: No bootstrap required, opened a new log
I20260806 10:59:09.556041 21494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:09.530633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56989 (local address 127.30.218.62:35965)
0806 10:59:09.531513 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:09.531529 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:09.531579 (+ 50us) server_negotiation.cc:407] Connection header received
0806 10:59:09.533844 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:09.533875 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:09.534457 (+ 582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:09.534932 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:09.536472 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:09.538619 (+ 2147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:09.540093 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:09.541205 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:09.541409 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:09.546167 (+ 4758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:09.546180 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:09.546201 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:09.546271 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:09.550408 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:09.551532 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:09.551542 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:09.551551 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:09.551827 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:09.552419 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:09.552423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:09.552425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:09.553186 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:09.553576 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:09.554111 (+ 535us) server_negotiation.cc:299] Negotiation successful
0806 10:59:09.554527 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":357,"threads_started":1}
I20260806 10:59:09.560431 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:59:09.563881 21491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 353baacb04dd4a1eb67307ee9ad06f7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353baacb04dd4a1eb67307ee9ad06f7a" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 35965 } }
I20260806 10:59:09.564343 21491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 353baacb04dd4a1eb67307ee9ad06f7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:59:09.564459 21491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 353baacb04dd4a1eb67307ee9ad06f7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353baacb04dd4a1eb67307ee9ad06f7a, State: Initialized, Role: FOLLOWER
I20260806 10:59:09.565202 21491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 353baacb04dd4a1eb67307ee9ad06f7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353baacb04dd4a1eb67307ee9ad06f7a" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 35965 } }
I20260806 10:59:09.565604 21491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 353baacb04dd4a1eb67307ee9ad06f7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:59:09.565773 21491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 353baacb04dd4a1eb67307ee9ad06f7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:59:09.565985 21491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 353baacb04dd4a1eb67307ee9ad06f7a [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:59:09.569170 21491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 353baacb04dd4a1eb67307ee9ad06f7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353baacb04dd4a1eb67307ee9ad06f7a" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 35965 } }
I20260806 10:59:09.569921 21491 leader_election.cc:302] T 00000000000000000000000000000000 P 353baacb04dd4a1eb67307ee9ad06f7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 353baacb04dd4a1eb67307ee9ad06f7a; no voters:
I20260806 10:59:09.570652 21491 leader_election.cc:288] T 00000000000000000000000000000000 P 353baacb04dd4a1eb67307ee9ad06f7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:59:09.570766 21501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 353baacb04dd4a1eb67307ee9ad06f7a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:59:09.572964 21491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 353baacb04dd4a1eb67307ee9ad06f7a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:59:09.575855 21501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 353baacb04dd4a1eb67307ee9ad06f7a [term 1 LEADER]: Becoming Leader. State: Replica: 353baacb04dd4a1eb67307ee9ad06f7a, State: Running, Role: LEADER
I20260806 10:59:09.576519 21501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 353baacb04dd4a1eb67307ee9ad06f7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353baacb04dd4a1eb67307ee9ad06f7a" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 35965 } }
I20260806 10:59:09.584194 21494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:09.565008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35115 (local address 127.30.218.62:35965)
0806 10:59:09.565234 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:09.565241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:09.565874 (+ 633us) server_negotiation.cc:407] Connection header received
0806 10:59:09.566108 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:09.566114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:09.566197 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:09.566280 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:09.568217 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:09.568999 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:09.570901 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:09.571758 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:09.571909 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:09.575794 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:09.575801 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:09.575806 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:09.575861 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:09.580355 (+ 4494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:09.581590 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:09.581597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:09.581601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:09.581787 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:09.582319 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:09.582325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:09.582329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:09.582770 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:09.583111 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:09.583527 (+ 416us) server_negotiation.cc:299] Negotiation successful
0806 10:59:09.583750 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 10:59:09.588413 21503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353baacb04dd4a1eb67307ee9ad06f7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 353baacb04dd4a1eb67307ee9ad06f7a. Latest consensus state: current_term: 1 leader_uuid: "353baacb04dd4a1eb67307ee9ad06f7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353baacb04dd4a1eb67307ee9ad06f7a" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 35965 } } }
I20260806 10:59:09.588346 21502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353baacb04dd4a1eb67307ee9ad06f7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "353baacb04dd4a1eb67307ee9ad06f7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353baacb04dd4a1eb67307ee9ad06f7a" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 35965 } } }
I20260806 10:59:09.588836 21503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353baacb04dd4a1eb67307ee9ad06f7a [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:09.588837 21502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353baacb04dd4a1eb67307ee9ad06f7a [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:09.593715 21515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:59:09.598654 21515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:59:09.607481 21515 catalog_manager.cc:1347] Generated new cluster ID: 41bb7a1e89a14c45a2425dfadd762e1d
I20260806 10:59:09.607611 21515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:59:09.625947 21515 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:59:09.627389 21515 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:59:09.635543 21515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 353baacb04dd4a1eb67307ee9ad06f7a: Generated new TSK 0
I20260806 10:59:09.636502 21515 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:59:09.645488 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 21420
I20260806 10:59:09.647428 21485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:59:09.741993 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 21420
W20260806 10:59:09.760867 21405 connection.cc:535] client connection to 127.30.218.62:35965 recv error: Network error: failed to read from TLS socket (remote: 127.30.218.62:35965): Connection reset by peer (error 104)
2026-08-06T10:59:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2157 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:59:09 dist-test-slave-49h3 krb5kdc[21524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:59:09 dist-test-slave-49h3 krb5kdc[21524](info): set up 2 sockets
Aug 06 10:59:09 dist-test-slave-49h3 krb5kdc[21524](info): commencing operation
krb5kdc: starting...
W20260806 10:59:10.838809 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:59:11 dist-test-slave-49h3 krb5kdc[21524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:59:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:59:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:59:11.208353 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:42919
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:36663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:42919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:59:11.568423 21537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:59:11.568780 21537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:59:11.568894 21537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:59:11.576653 21537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:59:11.576794 21537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:59:11.576845 21537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:59:11.576908 21537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:59:11.576952 21537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:59:11.587373 21537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:36663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:42919
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:42919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:59:11.588713 21537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:59:11.590094 21537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:59:11.602421 21543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:11.602510 21542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:11.602454 21545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:59:11.603444 21537 server_base.cc:1034] running on GCE node
I20260806 10:59:11.604300 21537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:59:11.605942 21537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:59:11.607179 21537 hybrid_clock.cc:648] HybridClock initialized: now 1786013951607124 us; error 53 us; skew 500 ppm
I20260806 10:59:11.607549 21537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:59:11 dist-test-slave-49h3 krb5kdc[21524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013951, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:59:11.611814 21537 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:59:11.613515 21537 webserver.cc:460] Webserver started at http://127.30.218.62:46765/ using document root <none> and password file <none>
I20260806 10:59:11.614266 21537 fs_manager.cc:359] Metadata directory not provided
I20260806 10:59:11.614405 21537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:59:11.614727 21537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:59:11.618073 21537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "64f658fc64294ea4a891287eefb4dc2a"
format_stamp: "Formatted at 2026-08-06 10:59:11 on dist-test-slave-49h3"
server_key: "ff0c263cf6496df973151f10ac4de531"
server_key_iv: "12be2aa162682539084286e5678652e8"
server_key_version: "encryptionkey@0"
I20260806 10:59:11.618937 21537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "64f658fc64294ea4a891287eefb4dc2a"
format_stamp: "Formatted at 2026-08-06 10:59:11 on dist-test-slave-49h3"
server_key: "ff0c263cf6496df973151f10ac4de531"
server_key_iv: "12be2aa162682539084286e5678652e8"
server_key_version: "encryptionkey@0"
I20260806 10:59:11.624938 21537 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 10:59:11.629002 21552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:59:11.630643 21537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:59:11.630856 21537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "64f658fc64294ea4a891287eefb4dc2a"
format_stamp: "Formatted at 2026-08-06 10:59:11 on dist-test-slave-49h3"
server_key: "ff0c263cf6496df973151f10ac4de531"
server_key_iv: "12be2aa162682539084286e5678652e8"
server_key_version: "encryptionkey@0"
I20260806 10:59:11.631074 21537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:59:11.661605 21537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:59:11.665930 21537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:59:11.666249 21537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:59:11.684772 21537 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:42919
I20260806 10:59:11.684793 21604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:42919 every 8 connection(s)
I20260806 10:59:11.686379 21537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:59:11.686774 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 21537
I20260806 10:59:11.686937 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:59:11.687552 31592 external_mini_cluster.cc:1428] Setting key d5260c16dc6347d3593f353a8667cf1b
I20260806 10:59:11.691319 21605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:59:11.701681 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:11.689695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40761 (local address 127.30.218.62:42919)
0806 10:59:11.690594 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:11.690607 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:11.690651 (+ 44us) server_negotiation.cc:407] Connection header received
0806 10:59:11.692453 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:11.692478 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:11.692901 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:11.693287 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:11.693988 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:11.695375 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:11.696771 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:11.697542 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:11.697731 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:11.699067 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:11.699075 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:11.699094 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:59:11.699700 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:11.699800 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:59:11.699834 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:11.699995 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:11.700318 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 10:59:11.700523 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":411,"threads_started":1}
I20260806 10:59:11.703025 21605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64f658fc64294ea4a891287eefb4dc2a: Bootstrap starting.
I20260806 10:59:11.706761 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:59:11.707134 21605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64f658fc64294ea4a891287eefb4dc2a: Neither blocks nor log segments found. Creating new log.
I20260806 10:59:11.708280 21605 log.cc:824] T 00000000000000000000000000000000 P 64f658fc64294ea4a891287eefb4dc2a: Log is configured to *not* fsync() on all Append() calls
I20260806 10:59:11.712280 21605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64f658fc64294ea4a891287eefb4dc2a: No bootstrap required, opened a new log
I20260806 10:59:11.719676 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:11.711633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43575 (local address 127.30.218.62:42919)
0806 10:59:11.711871 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:11.711878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:11.712307 (+ 429us) server_negotiation.cc:407] Connection header received
0806 10:59:11.712576 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:11.712581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:11.712661 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:11.712754 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:11.713430 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:11.714447 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:11.716521 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:11.717022 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:11.717137 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:11.718731 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:11.718737 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:11.718743 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:59:11.718770 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:11.718824 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:59:11.718843 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:11.718950 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:11.719222 (+ 272us) server_negotiation.cc:299] Negotiation successful
0806 10:59:11.719307 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 10:59:11.722563 21605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64f658fc64294ea4a891287eefb4dc2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64f658fc64294ea4a891287eefb4dc2a" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42919 } }
I20260806 10:59:11.723019 21605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64f658fc64294ea4a891287eefb4dc2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:59:11.723121 21605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64f658fc64294ea4a891287eefb4dc2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f658fc64294ea4a891287eefb4dc2a, State: Initialized, Role: FOLLOWER
I20260806 10:59:11.723753 21605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64f658fc64294ea4a891287eefb4dc2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64f658fc64294ea4a891287eefb4dc2a" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42919 } }
I20260806 10:59:11.724014 21605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64f658fc64294ea4a891287eefb4dc2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:59:11.724155 21605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64f658fc64294ea4a891287eefb4dc2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:59:11.724331 21605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64f658fc64294ea4a891287eefb4dc2a [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:59:11.726923 21605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64f658fc64294ea4a891287eefb4dc2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64f658fc64294ea4a891287eefb4dc2a" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42919 } }
I20260806 10:59:11.727548 21605 leader_election.cc:302] T 00000000000000000000000000000000 P 64f658fc64294ea4a891287eefb4dc2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64f658fc64294ea4a891287eefb4dc2a; no voters:
I20260806 10:59:11.728188 21605 leader_election.cc:288] T 00000000000000000000000000000000 P 64f658fc64294ea4a891287eefb4dc2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:59:11.728340 21616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64f658fc64294ea4a891287eefb4dc2a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:59:11.729533 21616 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64f658fc64294ea4a891287eefb4dc2a [term 1 LEADER]: Becoming Leader. State: Replica: 64f658fc64294ea4a891287eefb4dc2a, State: Running, Role: LEADER
I20260806 10:59:11.730238 21616 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64f658fc64294ea4a891287eefb4dc2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64f658fc64294ea4a891287eefb4dc2a" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42919 } }
I20260806 10:59:11.730686 21605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64f658fc64294ea4a891287eefb4dc2a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:59:11.733331 21617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64f658fc64294ea4a891287eefb4dc2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64f658fc64294ea4a891287eefb4dc2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64f658fc64294ea4a891287eefb4dc2a" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42919 } } }
I20260806 10:59:11.733850 21617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64f658fc64294ea4a891287eefb4dc2a [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:11.733676 21618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64f658fc64294ea4a891287eefb4dc2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64f658fc64294ea4a891287eefb4dc2a. Latest consensus state: current_term: 1 leader_uuid: "64f658fc64294ea4a891287eefb4dc2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64f658fc64294ea4a891287eefb4dc2a" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 42919 } } }
I20260806 10:59:11.734046 21618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64f658fc64294ea4a891287eefb4dc2a [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:11.736045 21625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:59:11.741250 21625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:59:11.751814 21625 catalog_manager.cc:1347] Generated new cluster ID: 71c828620d80417992fc15f5e89d920d
I20260806 10:59:11.752178 21625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:59:11.762010 21625 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:59:11.764103 21625 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:59:11.775837 21625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64f658fc64294ea4a891287eefb4dc2a: Generated new TSK 0
I20260806 10:59:11.777092 21625 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:59:11.788168 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 21537
W20260806 10:59:11.789829 21600 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40761
W20260806 10:59:11.790767 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 21537: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:59:11.790971 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 21537
W20260806 10:59:11.809353 21522 connection.cc:535] client connection to 127.30.218.62:42919 recv error: Network error: failed to read from TLS socket (remote: 127.30.218.62:42919): Connection reset by peer (error 104)
2026-08-06T10:59:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2048 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:59:11 dist-test-slave-49h3 krb5kdc[21639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:59:11 dist-test-slave-49h3 krb5kdc[21639](info): set up 2 sockets
Aug 06 10:59:11 dist-test-slave-49h3 krb5kdc[21639](info): commencing operation
krb5kdc: starting...
W20260806 10:59:12.888213 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:59:13 dist-test-slave-49h3 krb5kdc[21639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:59:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:59:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:59:13.248521 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:33143
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:43423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:33143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:59:13.604923 21652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:59:13.605273 21652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:59:13.605401 21652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:59:13.613050 21652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:59:13.613196 21652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:59:13.613255 21652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:59:13.613344 21652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:59:13.613400 21652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:59:13.624347 21652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:43423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:33143
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:33143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:59:13.625835 21652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:59:13.627271 21652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:59:13.640064 21661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:13.640105 21658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:13.640166 21657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:59:13.640930 21652 server_base.cc:1034] running on GCE node
I20260806 10:59:13.641857 21652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:59:13.643447 21652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:59:13.644677 21652 hybrid_clock.cc:648] HybridClock initialized: now 1786013953644636 us; error 52 us; skew 500 ppm
I20260806 10:59:13.645052 21652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:59:13 dist-test-slave-49h3 krb5kdc[21639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013953, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:59:13.648454 21652 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:59:13.650223 21652 webserver.cc:460] Webserver started at http://127.30.218.62:35661/ using document root <none> and password file <none>
I20260806 10:59:13.650997 21652 fs_manager.cc:359] Metadata directory not provided
I20260806 10:59:13.651130 21652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:59:13.651453 21652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:59:13.654832 21652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "93917eb4f999403d90e1320ca9703961"
format_stamp: "Formatted at 2026-08-06 10:59:13 on dist-test-slave-49h3"
server_key: "34b74377ef7bb1502818e17941ac3830"
server_key_iv: "6ff1d854a29d7a530f05d764351e7a9e"
server_key_version: "encryptionkey@0"
I20260806 10:59:13.655725 21652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "93917eb4f999403d90e1320ca9703961"
format_stamp: "Formatted at 2026-08-06 10:59:13 on dist-test-slave-49h3"
server_key: "34b74377ef7bb1502818e17941ac3830"
server_key_iv: "6ff1d854a29d7a530f05d764351e7a9e"
server_key_version: "encryptionkey@0"
I20260806 10:59:13.661904 21652 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260806 10:59:13.665907 21668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:59:13.667443 21652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 10:59:13.667650 21652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "93917eb4f999403d90e1320ca9703961"
format_stamp: "Formatted at 2026-08-06 10:59:13 on dist-test-slave-49h3"
server_key: "34b74377ef7bb1502818e17941ac3830"
server_key_iv: "6ff1d854a29d7a530f05d764351e7a9e"
server_key_version: "encryptionkey@0"
I20260806 10:59:13.667887 21652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:59:13.674423 21652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:59:13.679236 21652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:59:13.679563 21652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:59:13.698654 21652 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:33143
I20260806 10:59:13.698694 21719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:33143 every 8 connection(s)
I20260806 10:59:13.700191 21652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:59:13.703509 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 21652
I20260806 10:59:13.703716 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:59:13.704336 31592 external_mini_cluster.cc:1428] Setting key 1e9d695dc5519b7a0232cb536b86121a
I20260806 10:59:13.704681 21720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:59:13.713811 21723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:13.706393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49161 (local address 127.30.218.62:33143)
0806 10:59:13.707232 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:13.707245 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:13.707292 (+ 47us) server_negotiation.cc:407] Connection header received
0806 10:59:13.709803 (+ 2511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:13.709832 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:13.710277 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:13.710746 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:13.711941 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:13.711950 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:13.711964 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:59:13.711985 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:13.712063 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:59:13.712086 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:13.712339 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:13.712484 (+ 145us) server_negotiation.cc:299] Negotiation successful
0806 10:59:13.712701 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":368,"threads_started":1}
I20260806 10:59:13.717846 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:59:13.718866 21720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93917eb4f999403d90e1320ca9703961: Bootstrap starting.
I20260806 10:59:13.722524 21720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93917eb4f999403d90e1320ca9703961: Neither blocks nor log segments found. Creating new log.
I20260806 10:59:13.724176 21720 log.cc:824] T 00000000000000000000000000000000 P 93917eb4f999403d90e1320ca9703961: Log is configured to *not* fsync() on all Append() calls
I20260806 10:59:13.725265 21723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:13.722596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38813 (local address 127.30.218.62:33143)
0806 10:59:13.722823 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:13.722829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:13.723189 (+ 360us) server_negotiation.cc:407] Connection header received
0806 10:59:13.723389 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:13.723393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:13.723454 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:13.723541 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:13.724531 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:13.724536 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:13.724539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:59:13.724555 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:13.724602 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:59:13.724617 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:13.724719 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:13.724883 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 10:59:13.724960 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 10:59:13.727689 21720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93917eb4f999403d90e1320ca9703961: No bootstrap required, opened a new log
I20260806 10:59:13.735718 21720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93917eb4f999403d90e1320ca9703961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93917eb4f999403d90e1320ca9703961" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 33143 } }
I20260806 10:59:13.736286 21720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93917eb4f999403d90e1320ca9703961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:59:13.736400 21720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93917eb4f999403d90e1320ca9703961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93917eb4f999403d90e1320ca9703961, State: Initialized, Role: FOLLOWER
I20260806 10:59:13.737085 21720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93917eb4f999403d90e1320ca9703961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93917eb4f999403d90e1320ca9703961" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 33143 } }
I20260806 10:59:13.737322 21720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93917eb4f999403d90e1320ca9703961 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:59:13.737430 21720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93917eb4f999403d90e1320ca9703961 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:59:13.737640 21720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93917eb4f999403d90e1320ca9703961 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:59:13.739969 21720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93917eb4f999403d90e1320ca9703961 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93917eb4f999403d90e1320ca9703961" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 33143 } }
I20260806 10:59:13.740509 21720 leader_election.cc:302] T 00000000000000000000000000000000 P 93917eb4f999403d90e1320ca9703961 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93917eb4f999403d90e1320ca9703961; no voters:
I20260806 10:59:13.741161 21720 leader_election.cc:288] T 00000000000000000000000000000000 P 93917eb4f999403d90e1320ca9703961 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:59:13.741302 21730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93917eb4f999403d90e1320ca9703961 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:59:13.742425 21730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93917eb4f999403d90e1320ca9703961 [term 1 LEADER]: Becoming Leader. State: Replica: 93917eb4f999403d90e1320ca9703961, State: Running, Role: LEADER
I20260806 10:59:13.742998 21730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93917eb4f999403d90e1320ca9703961 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93917eb4f999403d90e1320ca9703961" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 33143 } }
I20260806 10:59:13.743557 21720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93917eb4f999403d90e1320ca9703961 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:59:13.746053 21731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93917eb4f999403d90e1320ca9703961 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93917eb4f999403d90e1320ca9703961" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93917eb4f999403d90e1320ca9703961" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 33143 } } }
I20260806 10:59:13.746568 21731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93917eb4f999403d90e1320ca9703961 [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:13.746986 21733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93917eb4f999403d90e1320ca9703961 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93917eb4f999403d90e1320ca9703961. Latest consensus state: current_term: 1 leader_uuid: "93917eb4f999403d90e1320ca9703961" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93917eb4f999403d90e1320ca9703961" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 33143 } } }
I20260806 10:59:13.747341 21733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93917eb4f999403d90e1320ca9703961 [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:13.748901 21740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:59:13.753983 21740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:59:13.763401 21740 catalog_manager.cc:1347] Generated new cluster ID: 66939973acfc4969af3c519dff02ff38
I20260806 10:59:13.763550 21740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:59:13.794479 21740 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:59:13.795015 21740 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:59:13.806797 21740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93917eb4f999403d90e1320ca9703961: Generated new TSK 0
I20260806 10:59:13.808075 21740 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:59:13.825366 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 21652
W20260806 10:59:13.826646 21715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49161
W20260806 10:59:13.827467 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 21652: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:59:13.827610 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 21652
2026-08-06T10:59:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2037 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:59:13 dist-test-slave-49h3 krb5kdc[21752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:59:13 dist-test-slave-49h3 krb5kdc[21752](info): set up 2 sockets
Aug 06 10:59:13 dist-test-slave-49h3 krb5kdc[21752](info): commencing operation
krb5kdc: starting...
W20260806 10:59:14.919417 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:59:15 dist-test-slave-49h3 krb5kdc[21752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:59:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:59:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:59:15.278918 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:46137
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:36791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:46137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:59:15.637822 21765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:59:15.638166 21765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:59:15.638291 21765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:59:15.645890 21765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:59:15.646064 21765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:59:15.646150 21765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:59:15.646236 21765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:59:15.646301 21765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:59:15.656679 21765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:36791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:46137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:46137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:59:15.658176 21765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:59:15.659610 21765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:59:15.672026 21773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:15.672163 21771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:15.672194 21770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:59:15.673162 21765 server_base.cc:1034] running on GCE node
I20260806 10:59:15.674075 21765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:59:15.675714 21765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:59:15.676952 21765 hybrid_clock.cc:648] HybridClock initialized: now 1786013955676904 us; error 37 us; skew 500 ppm
I20260806 10:59:15.677340 21765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:59:15 dist-test-slave-49h3 krb5kdc[21752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013955, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:59:15.680755 21765 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:59:15.682461 21765 webserver.cc:460] Webserver started at http://127.30.218.62:44953/ using document root <none> and password file <none>
I20260806 10:59:15.683195 21765 fs_manager.cc:359] Metadata directory not provided
I20260806 10:59:15.683306 21765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:59:15.683593 21765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:59:15.687018 21765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "1c7cbe641a0b4d75abcbd55cb7f4ff9d"
format_stamp: "Formatted at 2026-08-06 10:59:15 on dist-test-slave-49h3"
server_key: "908d5f81be5ec08b962fe9855cde3e51"
server_key_iv: "acf140ec02c4ca3cf42eb17fda942e14"
server_key_version: "encryptionkey@0"
I20260806 10:59:15.687840 21765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "1c7cbe641a0b4d75abcbd55cb7f4ff9d"
format_stamp: "Formatted at 2026-08-06 10:59:15 on dist-test-slave-49h3"
server_key: "908d5f81be5ec08b962fe9855cde3e51"
server_key_iv: "acf140ec02c4ca3cf42eb17fda942e14"
server_key_version: "encryptionkey@0"
I20260806 10:59:15.693651 21765 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 10:59:15.697638 21780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:59:15.699169 21765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:59:15.699388 21765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "1c7cbe641a0b4d75abcbd55cb7f4ff9d"
format_stamp: "Formatted at 2026-08-06 10:59:15 on dist-test-slave-49h3"
server_key: "908d5f81be5ec08b962fe9855cde3e51"
server_key_iv: "acf140ec02c4ca3cf42eb17fda942e14"
server_key_version: "encryptionkey@0"
I20260806 10:59:15.699617 21765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:59:15.711800 21765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:59:15.716344 21765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:59:15.716639 21765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:59:15.735381 21765 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:46137
I20260806 10:59:15.735399 21838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:46137 every 8 connection(s)
I20260806 10:59:15.736845 21765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:59:15.740782 21839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:59:15.745589 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 21765
I20260806 10:59:15.745800 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:59:15.746384 31592 external_mini_cluster.cc:1428] Setting key baa775ab9474eaa1bc05c3af76f4147b
I20260806 10:59:15.753396 21839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c7cbe641a0b4d75abcbd55cb7f4ff9d: Bootstrap starting.
I20260806 10:59:15.757481 21839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c7cbe641a0b4d75abcbd55cb7f4ff9d: Neither blocks nor log segments found. Creating new log.
I20260806 10:59:15.758791 21839 log.cc:824] T 00000000000000000000000000000000 P 1c7cbe641a0b4d75abcbd55cb7f4ff9d: Log is configured to *not* fsync() on all Append() calls
Aug 06 10:59:15 dist-test-slave-49h3 krb5kdc[21752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:59:15.762603 21839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c7cbe641a0b4d75abcbd55cb7f4ff9d: No bootstrap required, opened a new log
I20260806 10:59:15.772357 21839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c7cbe641a0b4d75abcbd55cb7f4ff9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7cbe641a0b4d75abcbd55cb7f4ff9d" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46137 } }
I20260806 10:59:15.772900 21839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c7cbe641a0b4d75abcbd55cb7f4ff9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:59:15.773017 21839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c7cbe641a0b4d75abcbd55cb7f4ff9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7cbe641a0b4d75abcbd55cb7f4ff9d, State: Initialized, Role: FOLLOWER
I20260806 10:59:15.773186 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:15.748425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40855 (local address 127.30.218.62:46137)
0806 10:59:15.749237 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:15.749250 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:15.749296 (+ 46us) server_negotiation.cc:407] Connection header received
0806 10:59:15.751658 (+ 2362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:15.751688 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:15.752181 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:15.752560 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:15.754610 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:15.756033 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:15.757898 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:15.758668 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:15.758816 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:15.763793 (+ 4977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:15.763804 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:15.763822 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:15.763864 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:15.767476 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:15.768643 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:15.768654 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:15.768664 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:15.768947 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:15.769355 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:15.769361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:15.769364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:15.770299 (+ 935us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:15.770669 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:15.771147 (+ 478us) server_negotiation.cc:299] Negotiation successful
0806 10:59:15.771691 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":353,"threads_started":1}
I20260806 10:59:15.773782 21839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c7cbe641a0b4d75abcbd55cb7f4ff9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7cbe641a0b4d75abcbd55cb7f4ff9d" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46137 } }
I20260806 10:59:15.774039 21839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c7cbe641a0b4d75abcbd55cb7f4ff9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:59:15.774212 21839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c7cbe641a0b4d75abcbd55cb7f4ff9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:59:15.774391 21839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c7cbe641a0b4d75abcbd55cb7f4ff9d [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:59:15.777113 21839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c7cbe641a0b4d75abcbd55cb7f4ff9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7cbe641a0b4d75abcbd55cb7f4ff9d" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46137 } }
I20260806 10:59:15.777693 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:59:15.777755 21839 leader_election.cc:302] T 00000000000000000000000000000000 P 1c7cbe641a0b4d75abcbd55cb7f4ff9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c7cbe641a0b4d75abcbd55cb7f4ff9d; no voters:
I20260806 10:59:15.778435 21839 leader_election.cc:288] T 00000000000000000000000000000000 P 1c7cbe641a0b4d75abcbd55cb7f4ff9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:59:15.778550 21844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c7cbe641a0b4d75abcbd55cb7f4ff9d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:59:15.780743 21839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c7cbe641a0b4d75abcbd55cb7f4ff9d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:59:15.782087 21844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c7cbe641a0b4d75abcbd55cb7f4ff9d [term 1 LEADER]: Becoming Leader. State: Replica: 1c7cbe641a0b4d75abcbd55cb7f4ff9d, State: Running, Role: LEADER
I20260806 10:59:15.782804 21844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c7cbe641a0b4d75abcbd55cb7f4ff9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7cbe641a0b4d75abcbd55cb7f4ff9d" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46137 } }
I20260806 10:59:15.785622 21845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7cbe641a0b4d75abcbd55cb7f4ff9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c7cbe641a0b4d75abcbd55cb7f4ff9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7cbe641a0b4d75abcbd55cb7f4ff9d" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46137 } } }
I20260806 10:59:15.786111 21845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7cbe641a0b4d75abcbd55cb7f4ff9d [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:15.789532 21852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:59:15.789271 21846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7cbe641a0b4d75abcbd55cb7f4ff9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c7cbe641a0b4d75abcbd55cb7f4ff9d. Latest consensus state: current_term: 1 leader_uuid: "1c7cbe641a0b4d75abcbd55cb7f4ff9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7cbe641a0b4d75abcbd55cb7f4ff9d" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 46137 } } }
I20260806 10:59:15.790303 21846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7cbe641a0b4d75abcbd55cb7f4ff9d [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:15.796345 21852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:59:15.811327 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:15.791343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52277 (local address 127.30.218.62:46137)
0806 10:59:15.791539 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:15.791555 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:15.792179 (+ 624us) server_negotiation.cc:407] Connection header received
0806 10:59:15.792396 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:15.792404 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:15.792510 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:15.792632 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:15.794905 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:15.795996 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:15.797909 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:15.798589 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:15.798727 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:15.802252 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:15.802265 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:15.802269 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:15.802320 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:15.806288 (+ 3968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:15.807455 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:15.807464 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:15.807468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:15.807657 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:15.808157 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:15.808163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:15.808167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:15.808725 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:15.809001 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:15.810573 (+ 1572us) server_negotiation.cc:299] Negotiation successful
0806 10:59:15.810872 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 10:59:15.811592 21852 catalog_manager.cc:1347] Generated new cluster ID: 87319ae3b0414981ac3b2585b2de3823
I20260806 10:59:15.811784 21852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:59:15.835527 21852 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:59:15.837014 21852 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:59:15.852439 21852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c7cbe641a0b4d75abcbd55cb7f4ff9d: Generated new TSK 0
I20260806 10:59:15.853425 21852 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:59:15.864086 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 21765
I20260806 10:59:15.865500 21834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:59:15.965764 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 21765
2026-08-06T10:59:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2137 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:59:16 dist-test-slave-49h3 krb5kdc[21873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:59:16 dist-test-slave-49h3 krb5kdc[21873](info): set up 2 sockets
Aug 06 10:59:16 dist-test-slave-49h3 krb5kdc[21873](info): commencing operation
krb5kdc: starting...
W20260806 10:59:17.063357 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:59:17 dist-test-slave-49h3 krb5kdc[21873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:59:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:59:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:59:17.432706 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:37187
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:37187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:59:17.791288 21888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:59:17.791625 21888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:59:17.791762 21888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:59:17.799424 21888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:59:17.799568 21888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:59:17.799647 21888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:59:17.799746 21888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:59:17.799803 21888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:59:17.810448 21888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:37187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:37187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:59:17.811913 21888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:59:17.813396 21888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:59:17.826195 21893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:17.826224 21896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:17.826174 21894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:59:17.827363 21888 server_base.cc:1034] running on GCE node
I20260806 10:59:17.828265 21888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:59:17.829945 21888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:59:17.831166 21888 hybrid_clock.cc:648] HybridClock initialized: now 1786013957831120 us; error 43 us; skew 500 ppm
I20260806 10:59:17.831539 21888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:59:17 dist-test-slave-49h3 krb5kdc[21873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013957, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:59:17.834924 21888 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:59:17.836725 21888 webserver.cc:460] Webserver started at http://127.30.218.62:40223/ using document root <none> and password file <none>
I20260806 10:59:17.837569 21888 fs_manager.cc:359] Metadata directory not provided
I20260806 10:59:17.837720 21888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:59:17.838050 21888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:59:17.841594 21888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "3169f927a2784e668031edb0ecd33c75"
format_stamp: "Formatted at 2026-08-06 10:59:17 on dist-test-slave-49h3"
server_key: "4baf47f2924f01133ca3bc3713e4af2a"
server_key_iv: "10cade0b012cebf364e34b954c9b5425"
server_key_version: "encryptionkey@0"
I20260806 10:59:17.842540 21888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "3169f927a2784e668031edb0ecd33c75"
format_stamp: "Formatted at 2026-08-06 10:59:17 on dist-test-slave-49h3"
server_key: "4baf47f2924f01133ca3bc3713e4af2a"
server_key_iv: "10cade0b012cebf364e34b954c9b5425"
server_key_version: "encryptionkey@0"
I20260806 10:59:17.848743 21888 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 10:59:17.852901 21903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:59:17.854506 21888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 10:59:17.854727 21888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "3169f927a2784e668031edb0ecd33c75"
format_stamp: "Formatted at 2026-08-06 10:59:17 on dist-test-slave-49h3"
server_key: "4baf47f2924f01133ca3bc3713e4af2a"
server_key_iv: "10cade0b012cebf364e34b954c9b5425"
server_key_version: "encryptionkey@0"
I20260806 10:59:17.854948 21888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:59:17.875679 21888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:59:17.880467 21888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:59:17.880792 21888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:59:17.900238 21888 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:37187
I20260806 10:59:17.900260 21960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:37187 every 8 connection(s)
I20260806 10:59:17.901918 21888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:59:17.906351 21961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:59:17.908702 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 21888
I20260806 10:59:17.908908 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:59:17.909510 31592 external_mini_cluster.cc:1428] Setting key 61856dd8b8652b391689961d39ce8500
I20260806 10:59:17.920114 21961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3169f927a2784e668031edb0ecd33c75: Bootstrap starting.
I20260806 10:59:17.923681 21961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3169f927a2784e668031edb0ecd33c75: Neither blocks nor log segments found. Creating new log.
I20260806 10:59:17.924356 21964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:17.911570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33241 (local address 127.30.218.62:37187)
0806 10:59:17.912506 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:17.912519 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:17.912570 (+ 51us) server_negotiation.cc:407] Connection header received
0806 10:59:17.914454 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:17.914486 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:17.914968 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:17.915370 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:17.916095 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:17.917640 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:17.919517 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:17.920297 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:17.920494 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:17.921800 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:17.921809 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:17.921825 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:59:17.922439 (+ 614us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:17.922529 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:59:17.922558 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:17.922666 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:17.922935 (+ 269us) server_negotiation.cc:299] Negotiation successful
0806 10:59:17.923243 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":426,"threads_started":1}
I20260806 10:59:17.924909 21961 log.cc:824] T 00000000000000000000000000000000 P 3169f927a2784e668031edb0ecd33c75: Log is configured to *not* fsync() on all Append() calls
I20260806 10:59:17.928154 21961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3169f927a2784e668031edb0ecd33c75: No bootstrap required, opened a new log
I20260806 10:59:17.928458 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:59:17.938361 21961 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3169f927a2784e668031edb0ecd33c75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3169f927a2784e668031edb0ecd33c75" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 37187 } }
I20260806 10:59:17.938993 21961 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3169f927a2784e668031edb0ecd33c75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:59:17.939127 21961 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3169f927a2784e668031edb0ecd33c75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3169f927a2784e668031edb0ecd33c75, State: Initialized, Role: FOLLOWER
I20260806 10:59:17.939967 21961 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3169f927a2784e668031edb0ecd33c75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3169f927a2784e668031edb0ecd33c75" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 37187 } }
I20260806 10:59:17.940325 21961 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3169f927a2784e668031edb0ecd33c75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:59:17.940507 21961 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3169f927a2784e668031edb0ecd33c75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:59:17.940614 21964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:17.933119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43495 (local address 127.30.218.62:37187)
0806 10:59:17.933305 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:17.933310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:17.933747 (+ 437us) server_negotiation.cc:407] Connection header received
0806 10:59:17.933952 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:17.933956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:17.934032 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:17.934126 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:17.934840 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:17.935657 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:17.937005 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:17.937668 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:17.937799 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:17.939301 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:17.939311 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:17.939317 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:59:17.939345 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:17.939448 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:59:17.939475 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:17.939570 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:17.939801 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 10:59:17.939918 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 10:59:17.940793 21961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3169f927a2784e668031edb0ecd33c75 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:59:17.944636 21961 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3169f927a2784e668031edb0ecd33c75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3169f927a2784e668031edb0ecd33c75" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 37187 } }
I20260806 10:59:17.945349 21961 leader_election.cc:302] T 00000000000000000000000000000000 P 3169f927a2784e668031edb0ecd33c75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3169f927a2784e668031edb0ecd33c75; no voters:
I20260806 10:59:17.946046 21961 leader_election.cc:288] T 00000000000000000000000000000000 P 3169f927a2784e668031edb0ecd33c75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:59:17.946187 21972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3169f927a2784e668031edb0ecd33c75 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:59:17.948448 21961 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3169f927a2784e668031edb0ecd33c75 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:59:17.949808 21972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3169f927a2784e668031edb0ecd33c75 [term 1 LEADER]: Becoming Leader. State: Replica: 3169f927a2784e668031edb0ecd33c75, State: Running, Role: LEADER
I20260806 10:59:17.950546 21972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3169f927a2784e668031edb0ecd33c75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3169f927a2784e668031edb0ecd33c75" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 37187 } }
I20260806 10:59:17.953707 21974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3169f927a2784e668031edb0ecd33c75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3169f927a2784e668031edb0ecd33c75. Latest consensus state: current_term: 1 leader_uuid: "3169f927a2784e668031edb0ecd33c75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3169f927a2784e668031edb0ecd33c75" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 37187 } } }
I20260806 10:59:17.954203 21974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3169f927a2784e668031edb0ecd33c75 [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:17.954216 21973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3169f927a2784e668031edb0ecd33c75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3169f927a2784e668031edb0ecd33c75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3169f927a2784e668031edb0ecd33c75" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 37187 } } }
I20260806 10:59:17.954612 21973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3169f927a2784e668031edb0ecd33c75 [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:17.956712 21983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:59:17.962620 21983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:59:17.976652 21983 catalog_manager.cc:1347] Generated new cluster ID: a8150ac974d747f8a17fbdad0806e291
I20260806 10:59:17.976809 21983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:59:17.990289 21983 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:59:17.991824 21983 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:59:18.008270 21983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3169f927a2784e668031edb0ecd33c75: Generated new TSK 0
I20260806 10:59:18.009279 21983 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:59:18.023749 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 21888
W20260806 10:59:18.025919 21955 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33241
W20260806 10:59:18.026793 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 21888: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:59:18.026955 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 21888
2026-08-06T10:59:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2060 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:59:18 dist-test-slave-49h3 krb5kdc[21993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:59:18 dist-test-slave-49h3 krb5kdc[21993](info): set up 2 sockets
Aug 06 10:59:18 dist-test-slave-49h3 krb5kdc[21993](info): commencing operation
krb5kdc: starting...
W20260806 10:59:19.123241 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.003s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:59:19 dist-test-slave-49h3 krb5kdc[21993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:59:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:59:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:59:19.490877 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:34589
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:34905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:34589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:59:19.848663 22006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:59:19.849009 22006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:59:19.849152 22006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:59:19.856969 22006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:59:19.857125 22006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:59:19.857187 22006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:59:19.857276 22006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:59:19.857328 22006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:59:19.868227 22006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:34905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:34589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:34589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:59:19.869747 22006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:59:19.871160 22006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:59:19.883559 22011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:19.883584 22012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:19.883615 22014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:59:19.884712 22006 server_base.cc:1034] running on GCE node
I20260806 10:59:19.885672 22006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:59:19.887315 22006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:59:19.888571 22006 hybrid_clock.cc:648] HybridClock initialized: now 1786013959888517 us; error 37 us; skew 500 ppm
I20260806 10:59:19.888984 22006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:59:19 dist-test-slave-49h3 krb5kdc[21993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013959, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:59:19.896226 22006 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:59:19.898119 22006 webserver.cc:460] Webserver started at http://127.30.218.62:35541/ using document root <none> and password file <none>
I20260806 10:59:19.898871 22006 fs_manager.cc:359] Metadata directory not provided
I20260806 10:59:19.899004 22006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:59:19.899339 22006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:59:19.902696 22006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "95da70fbab10496caaaf46991be625f1"
format_stamp: "Formatted at 2026-08-06 10:59:19 on dist-test-slave-49h3"
server_key: "fa48e8a9fdaac4ce41d9b009887a29db"
server_key_iv: "a129a12eaefeaa529d5a6b507cc222f6"
server_key_version: "encryptionkey@0"
I20260806 10:59:19.903663 22006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "95da70fbab10496caaaf46991be625f1"
format_stamp: "Formatted at 2026-08-06 10:59:19 on dist-test-slave-49h3"
server_key: "fa48e8a9fdaac4ce41d9b009887a29db"
server_key_iv: "a129a12eaefeaa529d5a6b507cc222f6"
server_key_version: "encryptionkey@0"
I20260806 10:59:19.909677 22006 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 10:59:19.913569 22021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:59:19.915113 22006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:59:19.915316 22006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "95da70fbab10496caaaf46991be625f1"
format_stamp: "Formatted at 2026-08-06 10:59:19 on dist-test-slave-49h3"
server_key: "fa48e8a9fdaac4ce41d9b009887a29db"
server_key_iv: "a129a12eaefeaa529d5a6b507cc222f6"
server_key_version: "encryptionkey@0"
I20260806 10:59:19.915548 22006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:59:19.922010 22006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:59:19.926527 22006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:59:19.926860 22006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:59:19.948378 22006 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:34589
I20260806 10:59:19.948405 22073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:34589 every 8 connection(s)
I20260806 10:59:19.949990 22006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:59:19.954218 22074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:59:19.958025 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 22006
I20260806 10:59:19.958233 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:59:19.958832 31592 external_mini_cluster.cc:1428] Setting key d062c283d780eee46bf39a23a25003f1
I20260806 10:59:19.967761 22077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:19.961012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47649 (local address 127.30.218.62:34589)
0806 10:59:19.961974 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:19.961986 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:19.962033 (+ 47us) server_negotiation.cc:407] Connection header received
0806 10:59:19.963888 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:19.963919 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:19.964370 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:19.964821 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:19.965992 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:19.965999 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:19.966014 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:59:19.966042 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:19.966111 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:59:19.966136 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:19.966336 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:19.966498 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 10:59:19.966697 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":449,"threads_started":1}
I20260806 10:59:19.968449 22074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95da70fbab10496caaaf46991be625f1: Bootstrap starting.
I20260806 10:59:19.972198 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:59:19.972735 22074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95da70fbab10496caaaf46991be625f1: Neither blocks nor log segments found. Creating new log.
I20260806 10:59:19.974226 22074 log.cc:824] T 00000000000000000000000000000000 P 95da70fbab10496caaaf46991be625f1: Log is configured to *not* fsync() on all Append() calls
I20260806 10:59:19.978236 22074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95da70fbab10496caaaf46991be625f1: No bootstrap required, opened a new log
I20260806 10:59:19.980458 22077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:19.977654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34935 (local address 127.30.218.62:34589)
0806 10:59:19.977875 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:19.977883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:19.978379 (+ 496us) server_negotiation.cc:407] Connection header received
0806 10:59:19.978594 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:19.978597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:19.978655 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:19.978743 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:19.979771 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:19.979775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:19.979778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 10:59:19.979795 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:19.979841 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 10:59:19.979857 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:19.979955 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:19.980083 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 10:59:19.980136 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 10:59:19.987186 22074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95da70fbab10496caaaf46991be625f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da70fbab10496caaaf46991be625f1" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34589 } }
I20260806 10:59:19.987629 22074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95da70fbab10496caaaf46991be625f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 10:59:19.987711 22074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95da70fbab10496caaaf46991be625f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95da70fbab10496caaaf46991be625f1, State: Initialized, Role: FOLLOWER
I20260806 10:59:19.988291 22074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95da70fbab10496caaaf46991be625f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da70fbab10496caaaf46991be625f1" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34589 } }
I20260806 10:59:19.988515 22074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95da70fbab10496caaaf46991be625f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 10:59:19.988622 22074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95da70fbab10496caaaf46991be625f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 10:59:19.988759 22074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95da70fbab10496caaaf46991be625f1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:59:19.991304 22074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95da70fbab10496caaaf46991be625f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da70fbab10496caaaf46991be625f1" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34589 } }
I20260806 10:59:19.991953 22074 leader_election.cc:302] T 00000000000000000000000000000000 P 95da70fbab10496caaaf46991be625f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95da70fbab10496caaaf46991be625f1; no voters:
I20260806 10:59:19.992588 22074 leader_election.cc:288] T 00000000000000000000000000000000 P 95da70fbab10496caaaf46991be625f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 10:59:19.992725 22084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95da70fbab10496caaaf46991be625f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:59:19.993371 22084 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95da70fbab10496caaaf46991be625f1 [term 1 LEADER]: Becoming Leader. State: Replica: 95da70fbab10496caaaf46991be625f1, State: Running, Role: LEADER
I20260806 10:59:19.993990 22084 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95da70fbab10496caaaf46991be625f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da70fbab10496caaaf46991be625f1" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34589 } }
I20260806 10:59:19.994854 22074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95da70fbab10496caaaf46991be625f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:59:19.997277 22085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95da70fbab10496caaaf46991be625f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95da70fbab10496caaaf46991be625f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da70fbab10496caaaf46991be625f1" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34589 } } }
I20260806 10:59:19.997819 22085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95da70fbab10496caaaf46991be625f1 [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:19.998302 22086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95da70fbab10496caaaf46991be625f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95da70fbab10496caaaf46991be625f1. Latest consensus state: current_term: 1 leader_uuid: "95da70fbab10496caaaf46991be625f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da70fbab10496caaaf46991be625f1" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 34589 } } }
I20260806 10:59:19.998620 22086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95da70fbab10496caaaf46991be625f1 [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:20.000661 22092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:59:20.006739 22092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:59:20.017529 22092 catalog_manager.cc:1347] Generated new cluster ID: 1e1b4e43f0c9412aa9a4b3137f1a4d6a
I20260806 10:59:20.017669 22092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:59:20.034488 22092 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:59:20.035025 22092 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:59:20.043130 22092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95da70fbab10496caaaf46991be625f1: Generated new TSK 0
I20260806 10:59:20.044111 22092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:59:20.051291 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 22006
W20260806 10:59:20.052500 22069 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47649
W20260806 10:59:20.053249 31592 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 22006: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 10:59:20.053385 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 22006
2026-08-06T10:59:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2027 ms)
[----------] 18 tests from AuthTokenIssuingTest (37655 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:59:20 dist-test-slave-49h3 krb5kdc[22107](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:59:20 dist-test-slave-49h3 krb5kdc[22107](info): set up 2 sockets
Aug 06 10:59:20 dist-test-slave-49h3 krb5kdc[22107](info): commencing operation
krb5kdc: starting...
W20260806 10:59:21.151592 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.003s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:59:21 dist-test-slave-49h3 krb5kdc[22107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:59:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:59:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:59:21.521414 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:44097
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:42273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:44097,127.30.218.61:37593,127.30.218.60:35707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:59:21.881595 22121 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 10:59:21.882122 22121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:59:21.882270 22121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:59:21.882388 22121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:59:21.891645 22121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:59:21.891793 22121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:59:21.891886 22121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:59:21.891968 22121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:59:21.892055 22121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:59:21.904125 22121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:42273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:44097,127.30.218.61:37593,127.30.218.60:35707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:44097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:59:21.905794 22121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:59:21.907312 22121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:59:21.919611 22127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:21.919734 22129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:21.919793 22126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:59:21.920655 22121 server_base.cc:1034] running on GCE node
I20260806 10:59:21.921573 22121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:59:21.923177 22121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:59:21.924449 22121 hybrid_clock.cc:648] HybridClock initialized: now 1786013961924386 us; error 48 us; skew 500 ppm
I20260806 10:59:21.924844 22121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:59:21 dist-test-slave-49h3 krb5kdc[22107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013961, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:59:21.928323 22121 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:59:21.929965 22121 webserver.cc:460] Webserver started at http://127.30.218.62:35201/ using document root <none> and password file <none>
I20260806 10:59:21.930723 22121 fs_manager.cc:359] Metadata directory not provided
I20260806 10:59:21.930845 22121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:59:21.931200 22121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:59:21.934502 22121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "f74e11d06eee415eb02d6dad25dc4bcb"
format_stamp: "Formatted at 2026-08-06 10:59:21 on dist-test-slave-49h3"
server_key: "7260df3025efaca1775ae11d61c2fec2"
server_key_iv: "2ab0fa34deaa8377aa44774abc4d6f83"
server_key_version: "encryptionkey@0"
I20260806 10:59:21.935571 22121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "f74e11d06eee415eb02d6dad25dc4bcb"
format_stamp: "Formatted at 2026-08-06 10:59:21 on dist-test-slave-49h3"
server_key: "7260df3025efaca1775ae11d61c2fec2"
server_key_iv: "2ab0fa34deaa8377aa44774abc4d6f83"
server_key_version: "encryptionkey@0"
I20260806 10:59:21.943253 22121 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260806 10:59:21.947247 22136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:59:21.948921 22121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 10:59:21.949127 22121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "f74e11d06eee415eb02d6dad25dc4bcb"
format_stamp: "Formatted at 2026-08-06 10:59:21 on dist-test-slave-49h3"
server_key: "7260df3025efaca1775ae11d61c2fec2"
server_key_iv: "2ab0fa34deaa8377aa44774abc4d6f83"
server_key_version: "encryptionkey@0"
I20260806 10:59:21.949381 22121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:59:21.963421 22121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:59:21.968257 22121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:59:21.968550 22121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:59:21.987582 22121 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:44097
I20260806 10:59:21.987604 22188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:44097 every 8 connection(s)
I20260806 10:59:21.989060 22121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:59:21.993114 22189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:59:21.998706 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 22121
I20260806 10:59:21.998930 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:59:21.999631 31592 external_mini_cluster.cc:1428] Setting key 584af51a0fc5868b5d70cb374be8d4e8
I20260806 10:59:22.000528 22189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.62" port: 44097 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:59:22 dist-test-slave-49h3 krb5kdc[22107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013961, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:59:22.028107 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.004170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:44097 (local address 127.0.0.1:44502)
0806 10:59:22.005236 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.005294 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 10:59:22.008016 (+ 2722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:59:22.009079 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:59:22.009086 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:59:22.009935 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:59:22.010457 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:22.010485 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:22.012245 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:22.012251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:22.014003 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:22.014022 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:22.015115 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:22.015123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:22.015300 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:22.016045 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:59:22.016076 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:59:22.018868 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:59:22.023462 (+ 4594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:22.023476 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:22.023488 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:22.023981 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:22.024533 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:22.024538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:22.024540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:22.024735 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:22.025893 (+ 1158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:59:22.025900 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:59:22.026185 (+ 285us) client_negotiation.cc:769] Sending connection context
0806 10:59:22.026416 (+ 231us) client_negotiation.cc:240] Negotiation successful
0806 10:59:22.026915 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":443,"threads_started":1}
I20260806 10:59:22.028153 22192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.004170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44502 (local address 127.30.218.62:44097)
0806 10:59:22.006423 (+ 2253us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:22.006429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:22.006482 (+ 53us) server_negotiation.cc:407] Connection header received
0806 10:59:22.008506 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:22.008512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:22.008810 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:22.008987 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:22.010747 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:22.012047 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:22.014209 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:22.014840 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:22.014994 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:22.019217 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:22.019230 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:22.019249 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:22.019295 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:22.023256 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:22.024160 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:22.024169 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:22.024173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:22.024388 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:22.024868 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:22.024872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:22.024875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:22.025736 (+ 861us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:22.025954 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:22.026612 (+ 658us) server_negotiation.cc:299] Negotiation successful
0806 10:59:22.027149 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":181,"server-negotiator.queue_time_us":2039,"spinlock_wait_cycles":58624,"thread_start_us":1741,"threads_started":1}
I20260806 10:59:22.031519 22189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:59:22.032809 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.032365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:37593 (local address 127.0.0.1:38878)
0806 10:59:22.032556 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.032678 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:37593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260806 10:59:22.033120 22137 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.218.61:37593: connect: Connection refused (error 111)
I20260806 10:59:22.034830 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.034437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:37593 (local address 127.0.0.1:38894)
0806 10:59:22.034593 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.034672 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:37593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 10:59:22.035063 22189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.61:37593: Network error: Client connection negotiation failed: client connection to 127.30.218.61:37593: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.30.218.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.61.keytab.
Entry for principal kudu/127.30.218.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.61.keytab.
I20260806 10:59:22.081611 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.61:37593
--webserver_interface=127.30.218.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.61.keytab
--principal=kudu/127.30.218.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:42273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:44097,127.30.218.61:37593,127.30.218.60:35707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:59:22.084424 22189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } attempt: 1
I20260806 10:59:22.085726 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.085148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:37593 (local address 127.0.0.1:38904)
0806 10:59:22.085382 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.085546 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:37593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 10:59:22.087075 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.086617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:37593 (local address 127.0.0.1:38916)
0806 10:59:22.086782 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.086878 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:37593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 10:59:22.087303 22189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.61:37593: Network error: Client connection negotiation failed: client connection to 127.30.218.61:37593: connect: Connection refused (error 111)
I20260806 10:59:22.155550 22189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } attempt: 2
I20260806 10:59:22.156715 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.156220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:37593 (local address 127.0.0.1:38926)
0806 10:59:22.156466 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.156567 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:37593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 10:59:22.157778 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.157345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:37593 (local address 127.0.0.1:38928)
0806 10:59:22.157536 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.157625 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:37593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 10:59:22.157986 22189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.61:37593: Network error: Client connection negotiation failed: client connection to 127.30.218.61:37593: connect: Connection refused (error 111)
I20260806 10:59:22.249256 22189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } attempt: 3
I20260806 10:59:22.250404 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.249953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:37593 (local address 127.0.0.1:38944)
0806 10:59:22.250180 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.250269 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:37593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 10:59:22.251384 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.251028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:37593 (local address 127.0.0.1:38950)
0806 10:59:22.251173 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.251243 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:37593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 10:59:22.251583 22189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.61:37593: Network error: Client connection negotiation failed: client connection to 127.30.218.61:37593: connect: Connection refused (error 111)
I20260806 10:59:22.394867 22189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } attempt: 4
I20260806 10:59:22.396237 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.395603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:37593 (local address 127.0.0.1:38956)
0806 10:59:22.395938 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.396074 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:37593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
I20260806 10:59:22.397509 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.397097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:37593 (local address 127.0.0.1:38958)
0806 10:59:22.397246 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.397328 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:37593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 10:59:22.397698 22189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.61:37593: Network error: Client connection negotiation failed: client connection to 127.30.218.61:37593: connect: Connection refused (error 111)
W20260806 10:59:22.437333 22198 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 10:59:22.437748 22198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:59:22.437861 22198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:59:22.437966 22198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:59:22.445729 22198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:59:22.445850 22198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:59:22.445910 22198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:59:22.445963 22198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:59:22.446043 22198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:59:22.456511 22198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:42273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:44097,127.30.218.61:37593,127.30.218.60:35707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.61.keytab
--principal=kudu/127.30.218.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.61:37593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.218.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:59:22.457937 22198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:59:22.459386 22198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:59:22.471746 22203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:22.471783 22204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:22.471814 22206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:59:22.472684 22198 server_base.cc:1034] running on GCE node
I20260806 10:59:22.473601 22198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:59:22.475160 22198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:59:22.476395 22198 hybrid_clock.cc:648] HybridClock initialized: now 1786013962476330 us; error 43 us; skew 500 ppm
I20260806 10:59:22.476773 22198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:59:22 dist-test-slave-49h3 krb5kdc[22107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013962, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:59:22.480255 22198 init.cc:377] Logged in from keytab as kudu/127.30.218.61@KRBTEST.COM (short username kudu)
I20260806 10:59:22.481973 22198 webserver.cc:460] Webserver started at http://127.30.218.61:40959/ using document root <none> and password file <none>
I20260806 10:59:22.482725 22198 fs_manager.cc:359] Metadata directory not provided
I20260806 10:59:22.482838 22198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:59:22.483178 22198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:59:22.486513 22198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-1/data/instance:
uuid: "bba0161fc3844f29a21f0448379ba675"
format_stamp: "Formatted at 2026-08-06 10:59:22 on dist-test-slave-49h3"
server_key: "e8e063cbc76ebc3ad649256ed636a0a6"
server_key_iv: "73c34a26f36fe1de89581b53134ceaaf"
server_key_version: "encryptionkey@0"
I20260806 10:59:22.487399 22198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-1/wal/instance:
uuid: "bba0161fc3844f29a21f0448379ba675"
format_stamp: "Formatted at 2026-08-06 10:59:22 on dist-test-slave-49h3"
server_key: "e8e063cbc76ebc3ad649256ed636a0a6"
server_key_iv: "73c34a26f36fe1de89581b53134ceaaf"
server_key_version: "encryptionkey@0"
I20260806 10:59:22.493172 22198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 10:59:22.497143 22213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:59:22.498639 22198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:59:22.498843 22198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-1/wal
uuid: "bba0161fc3844f29a21f0448379ba675"
format_stamp: "Formatted at 2026-08-06 10:59:22 on dist-test-slave-49h3"
server_key: "e8e063cbc76ebc3ad649256ed636a0a6"
server_key_iv: "73c34a26f36fe1de89581b53134ceaaf"
server_key_version: "encryptionkey@0"
I20260806 10:59:22.499079 22198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:59:22.519147 22198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:59:22.523679 22198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:59:22.523976 22198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:59:22.542814 22198 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.61:37593
I20260806 10:59:22.542829 22268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.61:37593 every 8 connection(s)
I20260806 10:59:22.544246 22198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-1/data/info.pb
I20260806 10:59:22.547930 22269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:59:22.550354 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 22198
I20260806 10:59:22.550539 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-1/wal/instance
I20260806 10:59:22.551095 31592 external_mini_cluster.cc:1428] Setting key c2ca49e1ed449610fc630f44fc1c8a8c
I20260806 10:59:22.556389 22269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.62" port: 44097 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:59:22 dist-test-slave-49h3 krb5kdc[22107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013962, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.61@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:59:22.581035 22271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.559643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44516 (local address 127.30.218.62:44097)
0806 10:59:22.560181 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:22.560188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:22.561242 (+ 1054us) server_negotiation.cc:407] Connection header received
0806 10:59:22.563410 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:22.563416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:22.563513 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:22.563647 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:22.565920 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:22.566906 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:22.569283 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:22.569772 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:22.569885 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:22.574475 (+ 4590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:22.574482 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:22.574485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:22.574547 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:22.577364 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:22.578516 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:22.578521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:22.578524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:22.578673 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:22.579164 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:22.579167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:22.579170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:22.579475 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:22.579606 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:22.580555 (+ 949us) server_negotiation.cc:299] Negotiation successful
0806 10:59:22.580753 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":259,"threads_started":1}
I20260806 10:59:22.582581 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.560242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:44097 (local address 127.0.0.1:44516)
0806 10:59:22.561039 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.561097 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 10:59:22.563053 (+ 1956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:59:22.564014 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:59:22.564028 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:59:22.564909 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:59:22.565651 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:22.565674 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:22.567096 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:22.567100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:22.569049 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:22.569068 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:22.570043 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:22.570049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:22.570395 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:22.571218 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:59:22.571256 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:59:22.574144 (+ 2888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:59:22.577583 (+ 3439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:22.577594 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:22.577615 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:22.578385 (+ 770us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:22.578783 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:22.578788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:22.578791 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:22.579063 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:22.579619 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:59:22.579629 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:59:22.580137 (+ 508us) client_negotiation.cc:769] Sending connection context
0806 10:59:22.580466 (+ 329us) client_negotiation.cc:240] Negotiation successful
0806 10:59:22.581018 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":315,"threads_started":1}
I20260806 10:59:22.584551 22269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.30.218.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.60@KRBTEST.COM" created.
Aug 06 10:59:22 dist-test-slave-49h3 krb5kdc[22107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013962, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.61@KRBTEST.COM for kudu/127.30.218.61@KRBTEST.COM
I20260806 10:59:22.600381 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.585390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:37593 (local address 127.0.0.1:38972)
0806 10:59:22.585601 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.585623 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 10:59:22.585726 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:59:22.586818 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:59:22.586822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:59:22.587314 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:59:22.587467 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:22.587477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:22.588919 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:22.588925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:22.589869 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:22.589883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:22.590873 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:22.590879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:22.591057 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:22.591655 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:59:22.591674 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:59:22.594290 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:59:22.597770 (+ 3480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:22.597776 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:22.597781 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:22.598174 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:22.598582 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:22.598588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:22.598592 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:22.598718 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:22.599381 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:59:22.599386 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:59:22.599544 (+ 158us) client_negotiation.cc:769] Sending connection context
0806 10:59:22.599662 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 10:59:22.599973 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 10:59:22.600490 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.585668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38972 (local address 127.30.218.61:37593)
0806 10:59:22.586374 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:22.586381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:22.586408 (+ 27us) server_negotiation.cc:407] Connection header received
0806 10:59:22.586455 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:22.586461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:22.586664 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:22.586811 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:22.587644 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:22.588772 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:22.590016 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:22.590721 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:22.590845 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:22.594543 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:22.594560 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:22.594576 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:22.594618 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:22.597623 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:22.598320 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:22.598328 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:22.598330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:22.598458 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:22.598832 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:22.598836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:22.598838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:22.599251 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:22.599472 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:22.599843 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 10:59:22.600106 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":550,"thread_start_us":416,"threads_started":1}
I20260806 10:59:22.603053 22269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:59:22.604306 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.603790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35707 (local address 127.0.0.1:36296)
0806 10:59:22.603990 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.604125 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260806 10:59:22.604575 22214 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
I20260806 10:59:22.606299 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.605914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35707 (local address 127.0.0.1:36302)
0806 10:59:22.606092 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.606182 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 10:59:22.606508 22269 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:35707: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.60.keytab.
Entry for principal kudu/127.30.218.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.60.keytab.
I20260806 10:59:22.633615 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.60:35707
--webserver_interface=127.30.218.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.60.keytab
--principal=kudu/127.30.218.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:42273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:44097,127.30.218.61:37593,127.30.218.60:35707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:59:22.655862 22269 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } attempt: 1
I20260806 10:59:22.657087 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.656615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35707 (local address 127.0.0.1:36306)
0806 10:59:22.656803 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.656916 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 10:59:22.658165 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.657798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35707 (local address 127.0.0.1:36318)
0806 10:59:22.657968 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.658047 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 10:59:22.658375 22269 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:35707: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
I20260806 10:59:22.696976 22189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } attempt: 5
Aug 06 10:59:22 dist-test-slave-49h3 krb5kdc[22107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013961, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for kudu/127.30.218.61@KRBTEST.COM
I20260806 10:59:22.712157 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.697768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:37593 (local address 127.0.0.1:38974)
0806 10:59:22.698037 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.698080 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 10:59:22.698243 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:59:22.698610 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:59:22.698616 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:59:22.699238 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:59:22.699528 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:22.699543 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:22.700579 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:22.700585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:22.701976 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:22.701994 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:22.702820 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:22.702827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:22.703069 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:22.703726 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:59:22.703751 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:59:22.706322 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:59:22.709531 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:22.709538 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:22.709543 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:22.709971 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:22.710376 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:22.710382 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:22.710387 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:22.710537 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:22.711106 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:59:22.711112 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:59:22.711275 (+ 163us) client_negotiation.cc:769] Sending connection context
0806 10:59:22.711405 (+ 130us) client_negotiation.cc:240] Negotiation successful
0806 10:59:22.711766 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260806 10:59:22.712150 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.697947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38974 (local address 127.30.218.61:37593)
0806 10:59:22.698131 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:22.698136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:22.698157 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:59:22.698379 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:22.698383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:22.698450 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:22.698535 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:22.699701 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:22.700417 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:22.702167 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:22.702657 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:22.702787 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:22.706564 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:22.706569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:22.706573 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:22.706621 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:22.709308 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:22.710111 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:22.710115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:22.710117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:22.710259 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:22.710662 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:22.710666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:22.710669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:22.710976 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:22.711118 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:22.711558 (+ 440us) server_negotiation.cc:299] Negotiation successful
0806 10:59:22.711808 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 10:59:22.713253 22189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:59:22.714298 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.713883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35707 (local address 127.0.0.1:36320)
0806 10:59:22.714051 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.714145 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 10:59:22.715253 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.714894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35707 (local address 127.0.0.1:36336)
0806 10:59:22.715036 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.715117 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 10:59:22.715659 22189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:35707: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
I20260806 10:59:22.726588 22269 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } attempt: 2
I20260806 10:59:22.727814 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.727335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35707 (local address 127.0.0.1:36350)
0806 10:59:22.727540 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.727643 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 10:59:22.729081 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.728692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35707 (local address 127.0.0.1:36366)
0806 10:59:22.728848 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.728936 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 10:59:22.729337 22269 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:35707: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
I20260806 10:59:22.766881 22189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } attempt: 1
I20260806 10:59:22.768187 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.767684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35707 (local address 127.0.0.1:36378)
0806 10:59:22.767894 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.768010 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 10:59:22.769199 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.768860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35707 (local address 127.0.0.1:36386)
0806 10:59:22.769010 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.769084 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 10:59:22.769474 22189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:35707: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
I20260806 10:59:22.820604 22269 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } attempt: 3
I20260806 10:59:22.821997 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.821412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35707 (local address 127.0.0.1:36400)
0806 10:59:22.821701 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.821819 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260806 10:59:22.823179 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.822709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35707 (local address 127.0.0.1:36412)
0806 10:59:22.822946 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.823042 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260806 10:59:22.823398 22269 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:35707: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
I20260806 10:59:22.837685 22189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } attempt: 2
I20260806 10:59:22.838896 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.838366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35707 (local address 127.0.0.1:36416)
0806 10:59:22.838614 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.838730 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260806 10:59:22.839867 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.839525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35707 (local address 127.0.0.1:36428)
0806 10:59:22.839679 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.839756 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 10:59:22.840087 22189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:35707: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
I20260806 10:59:22.946314 22189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } attempt: 3
I20260806 10:59:22.947427 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.946950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35707 (local address 127.0.0.1:36432)
0806 10:59:22.947157 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.947249 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 10:59:22.948449 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.948071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35707 (local address 127.0.0.1:36434)
0806 10:59:22.948254 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.948329 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 10:59:22.948648 22189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:35707: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
I20260806 10:59:22.966629 22269 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } attempt: 4
I20260806 10:59:22.967852 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.967346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35707 (local address 127.0.0.1:36446)
0806 10:59:22.967594 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.967686 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 10:59:22.968842 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:22.968503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35707 (local address 127.0.0.1:36452)
0806 10:59:22.968651 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:22.968724 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 10:59:22.969058 22269 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:35707: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35707: connect: Connection refused (error 111)
W20260806 10:59:23.007589 22278 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 10:59:23.007989 22278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:59:23.008090 22278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:59:23.008208 22278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:59:23.015936 22278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:59:23.016069 22278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:59:23.016156 22278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:59:23.016227 22278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:59:23.016316 22278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:59:23.026718 22278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:42273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:44097,127.30.218.61:37593,127.30.218.60:35707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.60.keytab
--principal=kudu/127.30.218.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.60:35707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.218.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:59:23.028285 22278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:59:23.029703 22278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:59:23.041543 22283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:23.041510 22284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:23.041555 22286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:59:23.042573 22278 server_base.cc:1034] running on GCE node
I20260806 10:59:23.043406 22278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:59:23.044867 22278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:59:23.046128 22278 hybrid_clock.cc:648] HybridClock initialized: now 1786013963046061 us; error 33 us; skew 500 ppm
I20260806 10:59:23.046478 22278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:59:23 dist-test-slave-49h3 krb5kdc[22107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013963, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:59:23.049676 22278 init.cc:377] Logged in from keytab as kudu/127.30.218.60@KRBTEST.COM (short username kudu)
I20260806 10:59:23.051163 22278 webserver.cc:460] Webserver started at http://127.30.218.60:45839/ using document root <none> and password file <none>
I20260806 10:59:23.051853 22278 fs_manager.cc:359] Metadata directory not provided
I20260806 10:59:23.051955 22278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:59:23.052273 22278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:59:23.055460 22278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-2/data/instance:
uuid: "a2f43b48ac4546ff94ee2fe2e61ec6a5"
format_stamp: "Formatted at 2026-08-06 10:59:23 on dist-test-slave-49h3"
server_key: "8a5d625ec84065a482320bb3913b2e12"
server_key_iv: "a838cfa3078a8c3c97eb007a891292d9"
server_key_version: "encryptionkey@0"
I20260806 10:59:23.056334 22278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-2/wal/instance:
uuid: "a2f43b48ac4546ff94ee2fe2e61ec6a5"
format_stamp: "Formatted at 2026-08-06 10:59:23 on dist-test-slave-49h3"
server_key: "8a5d625ec84065a482320bb3913b2e12"
server_key_iv: "a838cfa3078a8c3c97eb007a891292d9"
server_key_version: "encryptionkey@0"
I20260806 10:59:23.062212 22278 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 10:59:23.066251 22293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:59:23.067849 22278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 10:59:23.068063 22278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-2/wal
uuid: "a2f43b48ac4546ff94ee2fe2e61ec6a5"
format_stamp: "Formatted at 2026-08-06 10:59:23 on dist-test-slave-49h3"
server_key: "8a5d625ec84065a482320bb3913b2e12"
server_key_iv: "a838cfa3078a8c3c97eb007a891292d9"
server_key_version: "encryptionkey@0"
I20260806 10:59:23.068310 22278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:59:23.086683 22278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:59:23.091152 22278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:59:23.091461 22278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:59:23.111454 22278 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.60:35707
I20260806 10:59:23.111474 22345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.60:35707 every 8 connection(s)
I20260806 10:59:23.112978 22278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-2/data/info.pb
I20260806 10:59:23.117179 22346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:59:23.121939 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 22278
I20260806 10:59:23.122131 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786013589554914-31592-0/minicluster-data/master-2/wal/instance
I20260806 10:59:23.122563 31592 external_mini_cluster.cc:1428] Setting key a0774874e26a4f8ea8182199bb110438
I20260806 10:59:23.124938 22346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.62" port: 44097 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:59:23.125866 22189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } attempt: 4
Aug 06 10:59:23 dist-test-slave-49h3 krb5kdc[22107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013961, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for kudu/127.30.218.60@KRBTEST.COM
Aug 06 10:59:23 dist-test-slave-49h3 krb5kdc[22107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.60@KRBTEST.COM
Aug 06 10:59:23 dist-test-slave-49h3 krb5kdc[22107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013963, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.60@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:59:23.148943 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:23.126597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35707 (local address 127.0.0.1:36456)
0806 10:59:23.126804 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:23.126827 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:59:23.126962 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:59:23.129260 (+ 2298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:59:23.129268 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:59:23.129952 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:59:23.130204 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:23.130220 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.134298 (+ 4078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.134304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:23.135294 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:23.135308 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.136689 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.136697 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:23.136905 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:23.137631 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:59:23.137657 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:59:23.140383 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:59:23.145840 (+ 5457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:23.145847 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:23.145851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:23.146441 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:23.146957 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:23.146963 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:23.146966 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:23.147162 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:23.147817 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:59:23.147822 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:59:23.148008 (+ 186us) client_negotiation.cc:769] Sending connection context
0806 10:59:23.148222 (+ 214us) client_negotiation.cc:240] Negotiation successful
0806 10:59:23.148534 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 10:59:23.150468 22351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:23.128676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44530 (local address 127.30.218.62:44097)
0806 10:59:23.132783 (+ 4107us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:23.132787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:23.133658 (+ 871us) server_negotiation.cc:407] Connection header received
0806 10:59:23.133940 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:23.133945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:23.134039 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:23.134163 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:23.135752 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.136778 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.138677 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.139393 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.139555 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:23.143605 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:23.143612 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:23.143617 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:23.143662 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:23.147189 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:23.148081 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:23.148087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:23.148090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:23.148250 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:23.148734 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:23.148743 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:23.148746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:23.149125 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:23.149263 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:23.149829 (+ 566us) server_negotiation.cc:299] Negotiation successful
0806 10:59:23.150051 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4000,"thread_start_us":3861,"threads_started":1}
I20260806 10:59:23.150506 22349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:23.124736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36454 (local address 127.30.218.60:35707)
0806 10:59:23.125650 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:23.125664 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:23.125712 (+ 48us) server_negotiation.cc:407] Connection header received
0806 10:59:23.128234 (+ 2522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:23.128265 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:23.128784 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:23.131453 (+ 2669us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:23.131726 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.134091 (+ 2365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.135957 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.136494 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.136764 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:23.142006 (+ 5242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:23.142011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:23.142014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:23.142055 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:23.145384 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:23.146275 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:23.146283 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:23.146291 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:23.146446 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:23.146854 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:23.146859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:23.146861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:23.147618 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:23.147943 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:23.148405 (+ 462us) server_negotiation.cc:299] Negotiation successful
0806 10:59:23.148903 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":284,"server-negotiator.queue_time_us":629,"spinlock_wait_cycles":49792,"thread_start_us":394,"threads_started":1}
I20260806 10:59:23.150528 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:23.128820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:44097 (local address 127.0.0.1:44530)
0806 10:59:23.133492 (+ 4672us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:23.133541 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 10:59:23.133746 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:59:23.134262 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:59:23.134272 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:59:23.135210 (+ 938us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:59:23.135575 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:23.135589 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.137022 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.137028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:23.138481 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:23.138497 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.139686 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.139696 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:23.139888 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:23.140576 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:59:23.140610 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:59:23.143332 (+ 2722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:59:23.147329 (+ 3997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:23.147342 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:23.147349 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:23.147952 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:23.148388 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:23.148393 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:23.148396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:23.148587 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:23.149335 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:59:23.149341 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:59:23.149538 (+ 197us) client_negotiation.cc:769] Sending connection context
0806 10:59:23.149706 (+ 168us) client_negotiation.cc:240] Negotiation successful
0806 10:59:23.150067 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4382,"thread_start_us":4208,"threads_started":1}
I20260806 10:59:23.150564 22350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:23.126724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36456 (local address 127.30.218.60:35707)
0806 10:59:23.127622 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:23.127628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:23.127654 (+ 26us) server_negotiation.cc:407] Connection header received
0806 10:59:23.128250 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:23.128268 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:23.128786 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:23.130536 (+ 1750us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:23.131350 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.134093 (+ 2743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.135475 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.136502 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.136770 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:23.140692 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:23.140706 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:23.140725 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:23.140779 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:23.145629 (+ 4850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:23.146598 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:23.146603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:23.146607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:23.146783 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:23.147288 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:23.147292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:23.147295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:23.147663 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:23.149363 (+ 1700us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:23.149535 (+ 172us) server_negotiation.cc:299] Negotiation successful
0806 10:59:23.149763 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":141,"server-negotiator.queue_time_us":714,"spinlock_wait_cycles":11136,"thread_start_us":514,"threads_started":1}
I20260806 10:59:23.152599 22346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:59:23.159307 22189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb: Bootstrap starting.
Aug 06 10:59:23 dist-test-slave-49h3 krb5kdc[22107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013963, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.60@KRBTEST.COM for kudu/127.30.218.61@KRBTEST.COM
Aug 06 10:59:23 dist-test-slave-49h3 krb5kdc[22107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:59:23.163601 22189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb: Neither blocks nor log segments found. Creating new log.
I20260806 10:59:23.165072 22189 log.cc:824] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb: Log is configured to *not* fsync() on all Append() calls
I20260806 10:59:23.168783 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:23.153361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:37593 (local address 127.0.0.1:38986)
0806 10:59:23.153599 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:23.153617 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:59:23.153736 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:59:23.154044 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:59:23.154049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:59:23.154558 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:59:23.154765 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:23.154778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.156026 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.156030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:23.156969 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:23.156982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.158158 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.158164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:23.158349 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:23.159049 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:59:23.159073 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:59:23.161863 (+ 2790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:59:23.165624 (+ 3761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:23.165630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:23.165635 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:23.166251 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:23.166799 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:23.166802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:23.166805 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:23.167006 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:23.167714 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:59:23.167720 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:59:23.167904 (+ 184us) client_negotiation.cc:769] Sending connection context
0806 10:59:23.168078 (+ 174us) client_negotiation.cc:240] Negotiation successful
0806 10:59:23.168400 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 10:59:23.168871 22189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb: No bootstrap required, opened a new log
I20260806 10:59:23.168828 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:23.153382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38986 (local address 127.30.218.61:37593)
0806 10:59:23.153613 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:23.153617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:23.153704 (+ 87us) server_negotiation.cc:407] Connection header received
0806 10:59:23.153844 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:23.153850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:23.153925 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:23.154007 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:23.154947 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.155860 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.157121 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.157808 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.157934 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:23.162140 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:23.162158 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:23.162163 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:23.162214 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:23.165477 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:23.166420 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:23.166427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:23.166431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:23.166617 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:23.167144 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:23.167150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:23.167153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:23.167569 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:23.167731 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:23.168198 (+ 467us) server_negotiation.cc:299] Negotiation successful
0806 10:59:23.168451 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260806 10:59:23.170413 22346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:59:23.172672 22351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:23.154884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44532 (local address 127.30.218.62:44097)
0806 10:59:23.155071 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:23.155077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:23.155093 (+ 16us) server_negotiation.cc:407] Connection header received
0806 10:59:23.155153 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:23.155158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:23.155246 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:23.155347 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:23.156898 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.157790 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.159242 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.159734 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.159846 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:23.164741 (+ 4895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:23.164748 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:23.164752 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:23.164796 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:23.168285 (+ 3489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:23.169407 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:23.169411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:23.169413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:23.169620 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:23.170131 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:23.170137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:23.170140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:23.170561 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:23.170911 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:23.172038 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0806 10:59:23.172315 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Aug 06 10:59:23 dist-test-slave-49h3 krb5kdc[22107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013963, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.60@KRBTEST.COM for kudu/127.30.218.60@KRBTEST.COM
I20260806 10:59:23.179333 22189 raft_consensus.cc:348] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 44097 } } peers { permanent_uuid: "bba0161fc3844f29a21f0448379ba675" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } } peers { permanent_uuid: "a2f43b48ac4546ff94ee2fe2e61ec6a5" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } }
I20260806 10:59:23.179787 22189 raft_consensus.cc:729] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e11d06eee415eb02d6dad25dc4bcb, State: Initialized, Role: FOLLOWER
I20260806 10:59:23.180393 22189 consensus_queue.cc:260] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 44097 } } peers { permanent_uuid: "bba0161fc3844f29a21f0448379ba675" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } } peers { permanent_uuid: "a2f43b48ac4546ff94ee2fe2e61ec6a5" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } }
I20260806 10:59:23.181591 22354 sys_catalog.cc:455] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 44097 } } peers { permanent_uuid: "bba0161fc3844f29a21f0448379ba675" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } } peers { permanent_uuid: "a2f43b48ac4546ff94ee2fe2e61ec6a5" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } } }
I20260806 10:59:23.182181 22354 sys_catalog.cc:458] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:59:23.183144 22189 sys_catalog.cc:564] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb [sys.catalog]: configured and running, proceeding with master startup.
Aug 06 10:59:23 dist-test-slave-49h3 krb5kdc[22107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.61@KRBTEST.COM
I20260806 10:59:23.187362 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:23.171190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35707 (local address 127.0.0.1:36466)
0806 10:59:23.171566 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:23.171584 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:59:23.171713 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:59:23.172046 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:59:23.172051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:59:23.172579 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:59:23.172810 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:23.172825 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.174216 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.174222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:23.175381 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:23.175394 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.176504 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.176510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:23.176715 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:23.177405 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:59:23.177430 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:59:23.180068 (+ 2638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:59:23.184470 (+ 4402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:23.184476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:23.184480 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:23.185035 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:23.185558 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:23.185563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:23.185567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:23.185732 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:23.186300 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:59:23.186304 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:59:23.186452 (+ 148us) client_negotiation.cc:769] Sending connection context
0806 10:59:23.186572 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 10:59:23.186938 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":1}
I20260806 10:59:23.187403 22350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:23.171271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36466 (local address 127.30.218.60:35707)
0806 10:59:23.171465 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:23.171471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:23.171663 (+ 192us) server_negotiation.cc:407] Connection header received
0806 10:59:23.171821 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:23.171826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:23.171903 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:23.172002 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:23.173001 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.174075 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.175513 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.176160 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.176264 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:23.180306 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:23.180313 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:23.180317 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:23.180358 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:23.184268 (+ 3910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:23.185209 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:23.185215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:23.185218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:23.185372 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:23.185856 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:23.185860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:23.185863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:23.186184 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:23.186339 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:23.186680 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 10:59:23.187103 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":182,"server-negotiator.queue_time_us":61}
I20260806 10:59:23.193275 22346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2f43b48ac4546ff94ee2fe2e61ec6a5: Bootstrap starting.
I20260806 10:59:23.194190 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:23.175009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38990 (local address 127.30.218.61:37593)
0806 10:59:23.175197 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:23.175203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:23.175222 (+ 19us) server_negotiation.cc:407] Connection header received
0806 10:59:23.175277 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:23.175282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:23.175359 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:23.175457 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:23.177173 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.178435 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.180217 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.180656 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.181404 (+ 748us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:23.186049 (+ 4645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:23.186058 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:23.186064 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:23.186117 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:23.190397 (+ 4280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:23.191469 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:23.191476 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:23.191480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:23.191678 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:23.192221 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:23.192226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:23.192230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:23.192623 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:23.192894 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:23.193568 (+ 674us) server_negotiation.cc:299] Negotiation successful
0806 10:59:23.193839 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 10:59:23.197798 22346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2f43b48ac4546ff94ee2fe2e61ec6a5: Neither blocks nor log segments found. Creating new log.
I20260806 10:59:23.199355 22346 log.cc:824] T 00000000000000000000000000000000 P a2f43b48ac4546ff94ee2fe2e61ec6a5: Log is configured to *not* fsync() on all Append() calls
W20260806 10:59:23.200345 22365 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 10:59:23.200497 22365 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 10:59:23.208302 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:59:23.209772 22346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2f43b48ac4546ff94ee2fe2e61ec6a5: No bootstrap required, opened a new log
I20260806 10:59:23.212105 22163 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f74e11d06eee415eb02d6dad25dc4bcb"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44532
I20260806 10:59:23.212637 22163 raft_consensus.cc:482] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 10:59:23.212859 22163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:59:23.199568 22121 master_runner.cc:186] Error getting master registration for 127.30.218.61:37593: OK, instance_id { permanent_uuid: "bba0161fc3844f29a21f0448379ba675" instance_seqno: 1786013962537472 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 10:59:23.214691 22121 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 10:59:23.216766 22163 raft_consensus.cc:504] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 44097 } } peers { permanent_uuid: "bba0161fc3844f29a21f0448379ba675" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } } peers { permanent_uuid: "a2f43b48ac4546ff94ee2fe2e61ec6a5" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } }
I20260806 10:59:23.218638 22163 leader_election.cc:288] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb [CANDIDATE]: Term 1 election: Requested vote from peers bba0161fc3844f29a21f0448379ba675 (127.30.218.61:37593), a2f43b48ac4546ff94ee2fe2e61ec6a5 (127.30.218.60:35707)
I20260806 10:59:23.220785 22346 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2f43b48ac4546ff94ee2fe2e61ec6a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 44097 } } peers { permanent_uuid: "bba0161fc3844f29a21f0448379ba675" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } } peers { permanent_uuid: "a2f43b48ac4546ff94ee2fe2e61ec6a5" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } }
I20260806 10:59:23.221294 22346 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2f43b48ac4546ff94ee2fe2e61ec6a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f43b48ac4546ff94ee2fe2e61ec6a5, State: Initialized, Role: FOLLOWER
W20260806 10:59:23.221733 22142 tablet.cc:2369] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:59:23.223066 22346 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2f43b48ac4546ff94ee2fe2e61ec6a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 44097 } } peers { permanent_uuid: "bba0161fc3844f29a21f0448379ba675" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } } peers { permanent_uuid: "a2f43b48ac4546ff94ee2fe2e61ec6a5" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } }
I20260806 10:59:23.225252 22372 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2f43b48ac4546ff94ee2fe2e61ec6a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 44097 } } peers { permanent_uuid: "bba0161fc3844f29a21f0448379ba675" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } } peers { permanent_uuid: "a2f43b48ac4546ff94ee2fe2e61ec6a5" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } } }
I20260806 10:59:23.226410 22346 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2f43b48ac4546ff94ee2fe2e61ec6a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:59:23.226496 22320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a2f43b48ac4546ff94ee2fe2e61ec6a5"
I20260806 10:59:23.227061 22320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2f43b48ac4546ff94ee2fe2e61ec6a5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:59:23.232021 22320 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a2f43b48ac4546ff94ee2fe2e61ec6a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e11d06eee415eb02d6dad25dc4bcb in term 1.
I20260806 10:59:23.237888 22137 leader_election.cc:302] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f43b48ac4546ff94ee2fe2e61ec6a5, f74e11d06eee415eb02d6dad25dc4bcb; no voters:
I20260806 10:59:23.238612 22354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:59:23.241631 22372 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2f43b48ac4546ff94ee2fe2e61ec6a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:59:23.258306 22354 raft_consensus.cc:686] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb [term 1 LEADER]: Becoming Leader. State: Replica: f74e11d06eee415eb02d6dad25dc4bcb, State: Running, Role: LEADER
I20260806 10:59:23.259258 22354 consensus_queue.cc:237] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 44097 } } peers { permanent_uuid: "bba0161fc3844f29a21f0448379ba675" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } } peers { permanent_uuid: "a2f43b48ac4546ff94ee2fe2e61ec6a5" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } }
I20260806 10:59:23.268251 22269 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } attempt: 5
I20260806 10:59:23.276818 22376 sys_catalog.cc:455] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f74e11d06eee415eb02d6dad25dc4bcb. Latest consensus state: current_term: 1 leader_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 44097 } } peers { permanent_uuid: "bba0161fc3844f29a21f0448379ba675" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } } peers { permanent_uuid: "a2f43b48ac4546ff94ee2fe2e61ec6a5" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } } }
I20260806 10:59:23.277568 22376 sys_catalog.cc:458] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb [sys.catalog]: This master's current role is: LEADER
Aug 06 10:59:23 dist-test-slave-49h3 krb5kdc[22107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013962, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.61@KRBTEST.COM for kudu/127.30.218.60@KRBTEST.COM
I20260806 10:59:23.287185 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:23.268939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35707 (local address 127.0.0.1:36490)
0806 10:59:23.269349 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:23.269370 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:59:23.269527 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:59:23.270470 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:59:23.270475 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:59:23.271035 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:59:23.271269 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:23.271283 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.272876 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.272882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:23.274242 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:23.274259 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.275436 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.275442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:23.275682 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:23.276357 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:59:23.276385 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:59:23.279369 (+ 2984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:59:23.283865 (+ 4496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:23.283871 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:23.283876 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:23.284485 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:23.285012 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:23.285018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:23.285021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:23.285213 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:23.286074 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:59:23.286080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:59:23.286280 (+ 200us) client_negotiation.cc:769] Sending connection context
0806 10:59:23.286435 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 10:59:23.286760 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272}
I20260806 10:59:23.288314 22349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:23.269790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36490 (local address 127.30.218.60:35707)
0806 10:59:23.269966 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:23.269973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:23.269993 (+ 20us) server_negotiation.cc:407] Connection header received
0806 10:59:23.270059 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:23.270064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:23.270160 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:23.270262 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:23.271516 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.272667 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.274426 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.275073 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.275211 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:23.279647 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:23.279655 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:23.279660 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:23.279709 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:23.283652 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:23.284655 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:23.284661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:23.284665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:23.284848 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:23.285362 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:23.285367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:23.285370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:23.285819 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:23.285980 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:23.287635 (+ 1655us) server_negotiation.cc:299] Negotiation successful
0806 10:59:23.287918 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 10:59:23.291438 22351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:23.226308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44544 (local address 127.30.218.62:44097)
0806 10:59:23.226507 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:23.226514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:23.248127 (+ 21613us) server_negotiation.cc:407] Connection header received
0806 10:59:23.248486 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:23.248493 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:23.248607 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:23.248735 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:23.253790 (+ 5055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.255229 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.263172 (+ 7943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.264001 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.264122 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:23.280748 (+ 16626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:23.280755 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:23.280761 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:23.280823 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:23.284905 (+ 4082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:23.286938 (+ 2033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:23.286945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:23.286949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:23.287175 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:23.288649 (+ 1474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:23.288657 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:23.288660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:23.289137 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:23.289491 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:23.290701 (+ 1210us) server_negotiation.cc:299] Negotiation successful
0806 10:59:23.291002 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 10:59:23.291690 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:23.233309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38996 (local address 127.30.218.61:37593)
0806 10:59:23.233551 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:23.233558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:23.251640 (+ 18082us) server_negotiation.cc:407] Connection header received
0806 10:59:23.252502 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:23.252509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:23.252608 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:23.252702 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:23.262422 (+ 9720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.263610 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.271853 (+ 8243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.272532 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.272628 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:23.280442 (+ 7814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:23.280449 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:23.280454 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:23.280495 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:23.284042 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:23.287221 (+ 3179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:23.287228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:23.287230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:23.287394 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:23.288710 (+ 1316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:23.288716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:23.288719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:23.289099 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:23.289345 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:23.291010 (+ 1665us) server_negotiation.cc:299] Negotiation successful
0806 10:59:23.291303 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 10:59:23.294781 22269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bba0161fc3844f29a21f0448379ba675: Bootstrap starting.
I20260806 10:59:23.299530 22269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bba0161fc3844f29a21f0448379ba675: Neither blocks nor log segments found. Creating new log.
I20260806 10:59:23.301064 22269 log.cc:824] T 00000000000000000000000000000000 P bba0161fc3844f29a21f0448379ba675: Log is configured to *not* fsync() on all Append() calls
I20260806 10:59:23.305321 22350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:23.246584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36480 (local address 127.30.218.60:35707)
0806 10:59:23.246808 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:23.246815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:23.256944 (+ 10129us) server_negotiation.cc:407] Connection header received
0806 10:59:23.257735 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:23.257742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:23.257887 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:23.257972 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:23.268710 (+ 10738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.269959 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.272285 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:23.273061 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:23.273171 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:23.295369 (+ 22198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:23.295378 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:23.295382 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:23.295432 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:23.299690 (+ 4258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:23.301582 (+ 1892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:23.301589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:23.301592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:23.301828 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:23.302721 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:23.302728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:23.302732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:23.303151 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:23.303583 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:23.304620 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0806 10:59:23.304921 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 10:59:23.306102 22380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:59:23.308528 22380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:59:23.320554 22269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bba0161fc3844f29a21f0448379ba675: No bootstrap required, opened a new log
I20260806 10:59:23.333738 22269 raft_consensus.cc:348] T 00000000000000000000000000000000 P bba0161fc3844f29a21f0448379ba675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 44097 } } peers { permanent_uuid: "bba0161fc3844f29a21f0448379ba675" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } } peers { permanent_uuid: "a2f43b48ac4546ff94ee2fe2e61ec6a5" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } }
I20260806 10:59:23.334275 22269 raft_consensus.cc:729] T 00000000000000000000000000000000 P bba0161fc3844f29a21f0448379ba675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba0161fc3844f29a21f0448379ba675, State: Initialized, Role: FOLLOWER
I20260806 10:59:23.334933 22269 consensus_queue.cc:260] T 00000000000000000000000000000000 P bba0161fc3844f29a21f0448379ba675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 44097 } } peers { permanent_uuid: "bba0161fc3844f29a21f0448379ba675" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } } peers { permanent_uuid: "a2f43b48ac4546ff94ee2fe2e61ec6a5" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } }
I20260806 10:59:23.323458 22278 master_runner.cc:186] Error getting master registration for 127.30.218.61:37593: OK, instance_id { permanent_uuid: "bba0161fc3844f29a21f0448379ba675" instance_seqno: 1786013962537472 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 10:59:23.339578 22278 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 10:59:23.342301 22394 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba0161fc3844f29a21f0448379ba675 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 44097 } } peers { permanent_uuid: "bba0161fc3844f29a21f0448379ba675" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } } peers { permanent_uuid: "a2f43b48ac4546ff94ee2fe2e61ec6a5" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } } }
W20260806 10:59:23.343739 22299 tablet.cc:2369] T 00000000000000000000000000000000 P a2f43b48ac4546ff94ee2fe2e61ec6a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:59:23.343844 22269 sys_catalog.cc:564] T 00000000000000000000000000000000 P bba0161fc3844f29a21f0448379ba675 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:59:23.343917 22242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bba0161fc3844f29a21f0448379ba675"
I20260806 10:59:23.344452 22242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bba0161fc3844f29a21f0448379ba675 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:59:23.345417 22394 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba0161fc3844f29a21f0448379ba675 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:59:23.348160 22242 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bba0161fc3844f29a21f0448379ba675 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e11d06eee415eb02d6dad25dc4bcb in term 1.
W20260806 10:59:23.363034 22392 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2f43b48ac4546ff94ee2fe2e61ec6a5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 10:59:23.363752 22392 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 10:59:23.367888 22320 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a2f43b48ac4546ff94ee2fe2e61ec6a5 [term 1 FOLLOWER]: Refusing update from remote peer f74e11d06eee415eb02d6dad25dc4bcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:59:23.370419 22354 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f43b48ac4546ff94ee2fe2e61ec6a5" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:59:23.372113 22242 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bba0161fc3844f29a21f0448379ba675 [term 1 FOLLOWER]: Refusing update from remote peer f74e11d06eee415eb02d6dad25dc4bcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:59:23.373922 22354 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba0161fc3844f29a21f0448379ba675" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:59:23.392905 22394 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba0161fc3844f29a21f0448379ba675 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f74e11d06eee415eb02d6dad25dc4bcb. Latest consensus state: current_term: 1 leader_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 44097 } } peers { permanent_uuid: "bba0161fc3844f29a21f0448379ba675" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } } peers { permanent_uuid: "a2f43b48ac4546ff94ee2fe2e61ec6a5" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } } }
I20260806 10:59:23.393325 22394 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba0161fc3844f29a21f0448379ba675 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:59:23.398696 22372 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2f43b48ac4546ff94ee2fe2e61ec6a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f74e11d06eee415eb02d6dad25dc4bcb. Latest consensus state: current_term: 1 leader_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 44097 } } peers { permanent_uuid: "bba0161fc3844f29a21f0448379ba675" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } } peers { permanent_uuid: "a2f43b48ac4546ff94ee2fe2e61ec6a5" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } } }
I20260806 10:59:23.399098 22372 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2f43b48ac4546ff94ee2fe2e61ec6a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:59:23.403548 22396 mvcc.cc:205] Tried to move back new op lower bound from 7315513193915604992 to 7315513193518133248. Current Snapshot: MvccSnapshot[applied={T|T < 7315513193915604992}]
I20260806 10:59:23.410116 22376 sys_catalog.cc:455] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 44097 } } peers { permanent_uuid: "bba0161fc3844f29a21f0448379ba675" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } } peers { permanent_uuid: "a2f43b48ac4546ff94ee2fe2e61ec6a5" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } } }
I20260806 10:59:23.410531 22376 sys_catalog.cc:458] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:23.412832 22380 catalog_manager.cc:1347] Generated new cluster ID: 62f5ee42810f41459ecf4861030cc2fb
I20260806 10:59:23.412966 22380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:59:23.418987 22354 sys_catalog.cc:455] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 44097 } } peers { permanent_uuid: "bba0161fc3844f29a21f0448379ba675" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } } peers { permanent_uuid: "a2f43b48ac4546ff94ee2fe2e61ec6a5" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } } }
I20260806 10:59:23.420099 22354 sys_catalog.cc:458] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:23.420077 22402 mvcc.cc:205] Tried to move back new op lower bound from 7315513193915604992 to 7315513193518133248. Current Snapshot: MvccSnapshot[applied={T|T < 7315513193915604992}]
I20260806 10:59:23.426976 22394 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba0161fc3844f29a21f0448379ba675 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 44097 } } peers { permanent_uuid: "bba0161fc3844f29a21f0448379ba675" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } } peers { permanent_uuid: "a2f43b48ac4546ff94ee2fe2e61ec6a5" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } } }
I20260806 10:59:23.427345 22394 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba0161fc3844f29a21f0448379ba675 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:59:23.438179 22372 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2f43b48ac4546ff94ee2fe2e61ec6a5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e11d06eee415eb02d6dad25dc4bcb" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 44097 } } peers { permanent_uuid: "bba0161fc3844f29a21f0448379ba675" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 37593 } } peers { permanent_uuid: "a2f43b48ac4546ff94ee2fe2e61ec6a5" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35707 } } }
I20260806 10:59:23.438499 22372 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2f43b48ac4546ff94ee2fe2e61ec6a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:59:23.449210 22380 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:59:23.451938 22380 catalog_manager.cc:1504] Loading token signing keys...
I20260806 10:59:23.480176 22414 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 10:59:23.480917 22380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f74e11d06eee415eb02d6dad25dc4bcb: Generated new TSK 0
I20260806 10:59:23.482077 22380 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 10:59:23.580214 22351 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:23.576889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44556 (local address 127.30.218.62:44097)
0806 10:59:23.577263 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:23.577270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:23.578029 (+ 759us) server_negotiation.cc:407] Connection header received
0806 10:59:23.578376 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:23.578383 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:23.578468 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:23.578552 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:23.579907 (+ 1355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44556: BlockingRecv error: recv got EOF from 127.0.0.1:44556 (error 108)
Metrics: {"server-negotiator.queue_time_us":198}
W20260806 10:59:23.583114 22274 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:23.576822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39008 (local address 127.30.218.61:37593)
0806 10:59:23.577055 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:23.577062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:23.581139 (+ 4077us) server_negotiation.cc:407] Connection header received
0806 10:59:23.581467 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:23.581474 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:23.581575 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:23.581652 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:23.582842 (+ 1190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39008: BlockingRecv error: recv got EOF from 127.0.0.1:39008 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260806 10:59:23.586673 22350 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:23.584620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.30.218.60:35707)
0806 10:59:23.584801 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:23.584807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:23.585094 (+ 287us) server_negotiation.cc:407] Connection header received
0806 10:59:23.585258 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:23.585265 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:23.585336 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:23.585422 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:23.586362 (+ 940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36502: BlockingRecv error: recv got EOF from 127.0.0.1:36502 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 10:59:23.593762 22274 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:23.591103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39024 (local address 127.30.218.61:37593)
0806 10:59:23.591323 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:23.591330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:23.592181 (+ 851us) server_negotiation.cc:407] Connection header received
0806 10:59:23.592507 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:23.592512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:23.592613 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:23.592713 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:23.593518 (+ 805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39024: BlockingRecv error: recv got EOF from 127.0.0.1:39024 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260806 10:59:23.603569 22274 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:23.598798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39028 (local address 127.30.218.61:37593)
0806 10:59:23.598996 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:23.599003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:23.602021 (+ 3018us) server_negotiation.cc:407] Connection header received
0806 10:59:23.602292 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:23.602296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:23.602380 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:23.602478 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:23.603351 (+ 873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39028: BlockingRecv error: recv got EOF from 127.0.0.1:39028 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260806 10:59:23.611819 22274 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:23.608519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39032 (local address 127.30.218.61:37593)
0806 10:59:23.608713 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:23.608719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:23.609166 (+ 447us) server_negotiation.cc:407] Connection header received
0806 10:59:23.609384 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:23.609390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:23.609520 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:23.609610 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:23.610374 (+ 764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39032: BlockingRecv error: recv got EOF from 127.0.0.1:39032 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260806 10:59:23.625638 22274 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:23.623411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39036 (local address 127.30.218.61:37593)
0806 10:59:23.623671 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:23.623678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:23.624193 (+ 515us) server_negotiation.cc:407] Connection header received
0806 10:59:23.624397 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:23.624401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:23.624478 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:23.624565 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:23.625379 (+ 814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39036: BlockingRecv error: recv got EOF from 127.0.0.1:39036 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
W20260806 10:59:23.639707 22274 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:23.637515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39044 (local address 127.30.218.61:37593)
0806 10:59:23.637727 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:23.637735 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:23.638293 (+ 558us) server_negotiation.cc:407] Connection header received
0806 10:59:23.638486 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:23.638491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:23.638569 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:23.638659 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:23.639450 (+ 791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39044: BlockingRecv error: recv got EOF from 127.0.0.1:39044 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260806 10:59:23.661607 22274 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:23.659370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39052 (local address 127.30.218.61:37593)
0806 10:59:23.659585 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:23.659592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:23.660125 (+ 533us) server_negotiation.cc:407] Connection header received
0806 10:59:23.660320 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:23.660324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:23.660402 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:23.660490 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:23.661328 (+ 838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39052: BlockingRecv error: recv got EOF from 127.0.0.1:39052 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260806 10:59:23.706444 22274 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:23.703918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39060 (local address 127.30.218.61:37593)
0806 10:59:23.704112 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:23.704118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:23.704685 (+ 567us) server_negotiation.cc:407] Connection header received
0806 10:59:23.704884 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:23.704890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:23.705007 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:23.705125 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:23.706154 (+ 1029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39060: BlockingRecv error: recv got EOF from 127.0.0.1:39060 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260806 10:59:23.777398 22274 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:23.774964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39068 (local address 127.30.218.61:37593)
0806 10:59:23.775180 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:23.775187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:23.775754 (+ 567us) server_negotiation.cc:407] Connection header received
0806 10:59:23.775952 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:23.775957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:23.776066 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:23.776193 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:23.777154 (+ 961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39068: BlockingRecv error: recv got EOF from 127.0.0.1:39068 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260806 10:59:23.911087 22274 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:23.909112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.30.218.61:37593)
0806 10:59:23.909344 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:23.909351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:23.909634 (+ 283us) server_negotiation.cc:407] Connection header received
0806 10:59:23.909852 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:23.909856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:23.909929 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:23.910013 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:23.910827 (+ 814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39076: BlockingRecv error: recv got EOF from 127.0.0.1:39076 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260806 10:59:24.175160 22274 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:24.173185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39078 (local address 127.30.218.61:37593)
0806 10:59:24.173424 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:24.173431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:24.173710 (+ 279us) server_negotiation.cc:407] Connection header received
0806 10:59:24.173938 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:24.173942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:24.174023 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:24.174116 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:24.174923 (+ 807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39078: BlockingRecv error: recv got EOF from 127.0.0.1:39078 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 10:59:24.366106 22392 catalog_manager.cc:1259] Loaded cluster ID: 62f5ee42810f41459ecf4861030cc2fb
I20260806 10:59:24.366227 22392 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a2f43b48ac4546ff94ee2fe2e61ec6a5: loading cluster ID for follower catalog manager: success
I20260806 10:59:24.368894 22392 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a2f43b48ac4546ff94ee2fe2e61ec6a5: acquiring CA information for follower catalog manager: success
I20260806 10:59:24.370965 22392 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a2f43b48ac4546ff94ee2fe2e61ec6a5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 10:59:24.484899 22414 catalog_manager.cc:1259] Loaded cluster ID: 62f5ee42810f41459ecf4861030cc2fb
I20260806 10:59:24.485054 22414 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bba0161fc3844f29a21f0448379ba675: loading cluster ID for follower catalog manager: success
I20260806 10:59:24.488111 22414 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bba0161fc3844f29a21f0448379ba675: acquiring CA information for follower catalog manager: success
I20260806 10:59:24.490366 22414 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bba0161fc3844f29a21f0448379ba675: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 10:59:24.699334 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:24.691730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39092 (local address 127.30.218.61:37593)
0806 10:59:24.692407 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:24.692414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:24.692727 (+ 313us) server_negotiation.cc:407] Connection header received
0806 10:59:24.692968 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:24.692972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:24.693159 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:24.693280 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:59:24.693841 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:24.694732 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:24.696644 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:24.697136 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:24.697237 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:24.697943 (+ 706us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:59:24.698590 (+ 647us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:59:24.698719 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:24.698877 (+ 158us) server_negotiation.cc:299] Negotiation successful
0806 10:59:24.698959 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":308,"threads_started":1}
I20260806 10:59:24.718912 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:24.712669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36506 (local address 127.30.218.60:35707)
0806 10:59:24.713370 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:24.713376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:24.713400 (+ 24us) server_negotiation.cc:407] Connection header received
0806 10:59:24.713550 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:24.713556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:24.713728 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:24.713890 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:59:24.714360 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:24.715116 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:24.716451 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:24.716869 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:24.716988 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:24.717600 (+ 612us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:59:24.718213 (+ 613us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:59:24.718355 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:24.718508 (+ 153us) server_negotiation.cc:299] Negotiation successful
0806 10:59:24.718608 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":439,"threads_started":1}
I20260806 10:59:24.739265 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:24.726153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39100 (local address 127.30.218.61:37593)
0806 10:59:24.726417 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:24.726423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:24.730312 (+ 3889us) server_negotiation.cc:407] Connection header received
0806 10:59:24.730635 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:24.730642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:24.730751 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:24.730819 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:59:24.731682 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:24.733047 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:24.736803 (+ 3756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:24.737528 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:24.737652 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:24.738334 (+ 682us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:59:24.738471 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:59:24.738537 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:24.738796 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 10:59:24.738898 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260806 10:59:24.743772 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:24.726372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44560 (local address 127.30.218.62:44097)
0806 10:59:24.736048 (+ 9676us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:24.736054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:24.736070 (+ 16us) server_negotiation.cc:407] Connection header received
0806 10:59:24.736184 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:24.736188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:24.736361 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:24.736541 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:59:24.737308 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:24.738439 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:24.740537 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:24.741163 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:24.741292 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:24.742019 (+ 727us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:59:24.742538 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:59:24.742644 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:24.743343 (+ 699us) server_negotiation.cc:299] Negotiation successful
0806 10:59:24.743445 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9560,"thread_start_us":9442,"threads_started":1}
I20260806 10:59:24.746745 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:24.730791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36522 (local address 127.30.218.60:35707)
0806 10:59:24.731001 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:24.731007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:24.740325 (+ 9318us) server_negotiation.cc:407] Connection header received
0806 10:59:24.740637 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:24.740644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:24.740753 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:24.740815 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:59:24.741419 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:24.742481 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:24.744416 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:24.745008 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:24.745104 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:24.745849 (+ 745us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:59:24.745996 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:59:24.746092 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:24.746322 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 10:59:24.746421 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 10:59:24.749336 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 22121
I20260806 10:59:24.750494 22184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:59:24.866612 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 22121
I20260806 10:59:24.888957 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 22198
I20260806 10:59:24.890139 22264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:59:24.998534 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 22198
I20260806 10:59:25.019155 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 22278
I20260806 10:59:25.020368 22341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:59:25.132411 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 22278
2026-08-06T10:59:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5081 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 10:59:25 dist-test-slave-49h3 krb5kdc[22503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 10:59:25 dist-test-slave-49h3 krb5kdc[22503](info): set up 2 sockets
Aug 06 10:59:25 dist-test-slave-49h3 krb5kdc[22503](info): commencing operation
krb5kdc: starting...
W20260806 10:59:26.222952 31592 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.008s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/krb5kdc/test-user.keytab.
Aug 06 10:59:26 dist-test-slave-49h3 krb5kdc[22503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T10:59:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T10:59:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
Entry for principal kudu/127.30.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab.
I20260806 10:59:26.586359 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:40601
--webserver_interface=127.30.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:38547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:40601,127.30.218.61:33527,127.30.218.60:35999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 10:59:26.941810 22516 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 10:59:26.942191 22516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:59:26.942298 22516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:59:26.942413 22516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:59:26.950098 22516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:59:26.950217 22516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:59:26.950289 22516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:59:26.950363 22516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:59:26.950452 22516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:59:26.960960 22516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:38547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:40601,127.30.218.61:33527,127.30.218.60:35999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.62.keytab
--principal=kudu/127.30.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.62:40601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:59:26.962402 22516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:59:26.964017 22516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:59:26.976439 22521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:26.976480 22522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:26.976509 22524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:59:26.977392 22516 server_base.cc:1034] running on GCE node
I20260806 10:59:26.978330 22516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:59:26.979930 22516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:59:26.981165 22516 hybrid_clock.cc:648] HybridClock initialized: now 1786013966981085 us; error 61 us; skew 500 ppm
I20260806 10:59:26.981583 22516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:59:26 dist-test-slave-49h3 krb5kdc[22503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013966, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:59:26.984939 22516 init.cc:377] Logged in from keytab as kudu/127.30.218.62@KRBTEST.COM (short username kudu)
I20260806 10:59:26.986683 22516 webserver.cc:460] Webserver started at http://127.30.218.62:41727/ using document root <none> and password file <none>
I20260806 10:59:26.987442 22516 fs_manager.cc:359] Metadata directory not provided
I20260806 10:59:26.987564 22516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:59:26.987891 22516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:59:26.991138 22516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-0/data/instance:
uuid: "dea499cc79b3491ab5a5d13802e1a67c"
format_stamp: "Formatted at 2026-08-06 10:59:26 on dist-test-slave-49h3"
server_key: "fb1911b5840ad00996ca8fc1ef0e09a6"
server_key_iv: "942ad432ecc48479b85f88257af31df6"
server_key_version: "encryptionkey@0"
I20260806 10:59:26.992008 22516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-0/wal/instance:
uuid: "dea499cc79b3491ab5a5d13802e1a67c"
format_stamp: "Formatted at 2026-08-06 10:59:26 on dist-test-slave-49h3"
server_key: "fb1911b5840ad00996ca8fc1ef0e09a6"
server_key_iv: "942ad432ecc48479b85f88257af31df6"
server_key_version: "encryptionkey@0"
I20260806 10:59:26.997653 22516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260806 10:59:27.001417 22531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:59:27.002954 22516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:59:27.003144 22516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-0/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-0/wal
uuid: "dea499cc79b3491ab5a5d13802e1a67c"
format_stamp: "Formatted at 2026-08-06 10:59:26 on dist-test-slave-49h3"
server_key: "fb1911b5840ad00996ca8fc1ef0e09a6"
server_key_iv: "942ad432ecc48479b85f88257af31df6"
server_key_version: "encryptionkey@0"
I20260806 10:59:27.003376 22516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:59:27.020416 22516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:59:27.024897 22516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:59:27.025213 22516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:59:27.044986 22516 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.62:40601
I20260806 10:59:27.045006 22584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.62:40601 every 8 connection(s)
I20260806 10:59:27.046623 22516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-0/data/info.pb
I20260806 10:59:27.050710 22585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:59:27.051441 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 22516
I20260806 10:59:27.051645 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-0/wal/instance
I20260806 10:59:27.052323 31592 external_mini_cluster.cc:1428] Setting key d1333b9fae20fa23bce0a5ebc524238c
I20260806 10:59:27.060329 22585 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40601 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:59:27 dist-test-slave-49h3 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013966, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:59:27.087061 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.065976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36050 (local address 127.30.218.62:40601)
0806 10:59:27.066738 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:27.066745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:27.066800 (+ 55us) server_negotiation.cc:407] Connection header received
0806 10:59:27.068215 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:27.068223 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:27.068562 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:27.068802 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:27.070394 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:27.071689 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:27.073851 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:27.074548 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:27.074718 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:27.079272 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:27.079281 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:27.079296 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:27.079339 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:27.082339 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:27.083297 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:27.083304 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:27.083306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:27.083482 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:27.083930 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:27.083934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:27.083936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:27.084485 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:27.084670 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:27.085359 (+ 689us) server_negotiation.cc:299] Negotiation successful
0806 10:59:27.085788 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":357,"threads_started":1}
I20260806 10:59:27.087126 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.064445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:40601 (local address 127.0.0.1:36050)
0806 10:59:27.065406 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.065489 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0806 10:59:27.067763 (+ 2274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:59:27.068811 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:59:27.068816 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:59:27.069617 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:59:27.070143 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:27.070167 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:27.071864 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:27.071869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:27.073658 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:27.073683 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:27.074826 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:27.074833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:27.075021 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:27.075838 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:59:27.075883 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:59:27.078950 (+ 3067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:59:27.082514 (+ 3564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:27.082530 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:27.082544 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:27.083141 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:27.083601 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:27.083606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:27.083609 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:27.083827 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:27.084652 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:59:27.084662 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:59:27.084924 (+ 262us) client_negotiation.cc:769] Sending connection context
0806 10:59:27.085231 (+ 307us) client_negotiation.cc:240] Negotiation successful
0806 10:59:27.085975 (+ 744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":160,"thread_start_us":398,"threads_started":1}
WARNING: no policy specified for kudu/127.30.218.61@KRBTEST.COM; defaulting to no policy
I20260806 10:59:27.091138 22585 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:59:27.092475 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.091964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33527 (local address 127.0.0.1:36410)
0806 10:59:27.092203 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.092339 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260806 10:59:27.092806 22534 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33527: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.61@KRBTEST.COM" created.
I20260806 10:59:27.094723 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.094247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33527 (local address 127.0.0.1:36412)
0806 10:59:27.094404 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.094487 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 10:59:27.095009 22585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.61:33527: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33527: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.61.keytab.
Entry for principal kudu/127.30.218.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.61.keytab.
I20260806 10:59:27.136988 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.61:33527
--webserver_interface=127.30.218.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.61.keytab
--principal=kudu/127.30.218.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:38547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:40601,127.30.218.61:33527,127.30.218.60:35999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:59:27.144439 22585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } attempt: 1
I20260806 10:59:27.145821 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.145235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33527 (local address 127.0.0.1:36414)
0806 10:59:27.145492 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.145651 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 10:59:27.146925 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.146526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33527 (local address 127.0.0.1:36420)
0806 10:59:27.146679 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.146765 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 10:59:27.147171 22585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.61:33527: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33527: connect: Connection refused (error 111)
I20260806 10:59:27.215426 22585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } attempt: 2
I20260806 10:59:27.216718 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.216197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33527 (local address 127.0.0.1:36432)
0806 10:59:27.216454 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.216566 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260806 10:59:27.217780 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.217377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33527 (local address 127.0.0.1:36440)
0806 10:59:27.217564 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.217641 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 10:59:27.217978 22585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.61:33527: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33527: connect: Connection refused (error 111)
I20260806 10:59:27.309222 22585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } attempt: 3
I20260806 10:59:27.310381 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.309924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33527 (local address 127.0.0.1:36454)
0806 10:59:27.310146 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.310240 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 10:59:27.311422 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.311034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33527 (local address 127.0.0.1:36460)
0806 10:59:27.311188 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.311264 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 10:59:27.311614 22585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.61:33527: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33527: connect: Connection refused (error 111)
I20260806 10:59:27.454885 22585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } attempt: 4
I20260806 10:59:27.456167 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.455594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33527 (local address 127.0.0.1:36468)
0806 10:59:27.455917 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.456013 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260806 10:59:27.457177 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.456810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33527 (local address 127.0.0.1:36472)
0806 10:59:27.456960 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.457034 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 10:59:27.457365 22585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.61:33527: Network error: Client connection negotiation failed: client connection to 127.30.218.61:33527: connect: Connection refused (error 111)
W20260806 10:59:27.495461 22592 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 10:59:27.495877 22592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:59:27.495980 22592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:59:27.496104 22592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:59:27.503964 22592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:59:27.504096 22592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:59:27.504181 22592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:59:27.504247 22592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:59:27.504338 22592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:59:27.515223 22592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:38547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:40601,127.30.218.61:33527,127.30.218.60:35999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.61.keytab
--principal=kudu/127.30.218.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.61:33527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.218.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:59:27.516697 22592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:59:27.518153 22592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:59:27.530526 22601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:27.530696 22599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:27.530730 22598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:59:27.531613 22592 server_base.cc:1034] running on GCE node
I20260806 10:59:27.532480 22592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:59:27.534075 22592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:59:27.535320 22592 hybrid_clock.cc:648] HybridClock initialized: now 1786013967535252 us; error 39 us; skew 500 ppm
I20260806 10:59:27.535657 22592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:59:27 dist-test-slave-49h3 krb5kdc[22503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013967, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:59:27.539018 22592 init.cc:377] Logged in from keytab as kudu/127.30.218.61@KRBTEST.COM (short username kudu)
I20260806 10:59:27.540648 22592 webserver.cc:460] Webserver started at http://127.30.218.61:39897/ using document root <none> and password file <none>
I20260806 10:59:27.541406 22592 fs_manager.cc:359] Metadata directory not provided
I20260806 10:59:27.541549 22592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:59:27.541857 22592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:59:27.545156 22592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-1/data/instance:
uuid: "30b5b7cf19a94117ac993d27f6862fc5"
format_stamp: "Formatted at 2026-08-06 10:59:27 on dist-test-slave-49h3"
server_key: "614a5fc0c8886c7d14ac0dd8b3aeb564"
server_key_iv: "f645a9e9e953102f08bf5fb8661c7e66"
server_key_version: "encryptionkey@0"
I20260806 10:59:27.546145 22592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-1/wal/instance:
uuid: "30b5b7cf19a94117ac993d27f6862fc5"
format_stamp: "Formatted at 2026-08-06 10:59:27 on dist-test-slave-49h3"
server_key: "614a5fc0c8886c7d14ac0dd8b3aeb564"
server_key_iv: "f645a9e9e953102f08bf5fb8661c7e66"
server_key_version: "encryptionkey@0"
I20260806 10:59:27.552009 22592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 10:59:27.556066 22608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:59:27.557670 22592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 10:59:27.557857 22592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-1/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-1/wal
uuid: "30b5b7cf19a94117ac993d27f6862fc5"
format_stamp: "Formatted at 2026-08-06 10:59:27 on dist-test-slave-49h3"
server_key: "614a5fc0c8886c7d14ac0dd8b3aeb564"
server_key_iv: "f645a9e9e953102f08bf5fb8661c7e66"
server_key_version: "encryptionkey@0"
I20260806 10:59:27.558072 22592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:59:27.581627 22592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:59:27.586282 22592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:59:27.586661 22592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:59:27.605036 22592 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.61:33527
I20260806 10:59:27.605007 22660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.61:33527 every 8 connection(s)
I20260806 10:59:27.606660 22592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-1/data/info.pb
I20260806 10:59:27.610805 22661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:59:27.615399 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 22592
I20260806 10:59:27.615589 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-1/wal/instance
I20260806 10:59:27.616008 31592 external_mini_cluster.cc:1428] Setting key 4b6075eae2a246573e8627f299849f4e
I20260806 10:59:27.617605 22661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40601 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 10:59:27 dist-test-slave-49h3 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013967, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.61@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:59:27.638827 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.620226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36062 (local address 127.30.218.62:40601)
0806 10:59:27.620717 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:27.620722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:27.621805 (+ 1083us) server_negotiation.cc:407] Connection header received
0806 10:59:27.623672 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:27.623678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:27.623760 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:27.623855 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:27.625423 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:27.626237 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:27.627789 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:27.628216 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:27.628329 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:27.632289 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:27.632295 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:27.632298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:27.632341 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:27.635107 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:27.636223 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:27.636228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:27.636230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:27.636376 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:27.636901 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:27.636905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:27.636907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:27.637243 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:27.637384 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:27.638324 (+ 940us) server_negotiation.cc:299] Negotiation successful
0806 10:59:27.638535 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":265,"threads_started":1}
I20260806 10:59:27.640196 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.620865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:40601 (local address 127.0.0.1:36062)
0806 10:59:27.621651 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.621688 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 10:59:27.623401 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:59:27.624108 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:59:27.624117 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:59:27.624650 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:59:27.625209 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:27.625226 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:27.626400 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:27.626404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:27.627639 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:27.627653 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:27.628449 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:27.628457 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:27.628701 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:27.629393 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:59:27.629414 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:59:27.631963 (+ 2549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:59:27.635300 (+ 3337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:27.635311 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:27.635332 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:27.636068 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:27.636490 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:27.636494 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:27.636498 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:27.636748 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:27.637387 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:59:27.637395 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:59:27.637894 (+ 499us) client_negotiation.cc:769] Sending connection context
0806 10:59:27.638204 (+ 310us) client_negotiation.cc:240] Negotiation successful
0806 10:59:27.638749 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":320,"threads_started":1}
I20260806 10:59:27.642151 22661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.30.218.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.218.60@KRBTEST.COM" created.
Aug 06 10:59:27 dist-test-slave-49h3 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013967, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.61@KRBTEST.COM for kudu/127.30.218.61@KRBTEST.COM
I20260806 10:59:27.660156 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.643040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33527 (local address 127.0.0.1:36488)
0806 10:59:27.643225 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.643243 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 10:59:27.643345 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:59:27.644463 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:59:27.644467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:59:27.644932 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:59:27.645081 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:27.645090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:27.646646 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:27.646650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:27.647492 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:27.647504 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:27.648352 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:27.648358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:27.648518 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:27.649163 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:59:27.649179 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:59:27.653534 (+ 4355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:59:27.657668 (+ 4134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:27.657672 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:27.657675 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:27.658103 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:27.658523 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:27.658527 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:27.658530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:27.658671 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:27.659386 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:59:27.659389 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:59:27.659516 (+ 127us) client_negotiation.cc:769] Sending connection context
0806 10:59:27.659629 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 10:59:27.659862 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 10:59:27.660641 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.643223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36488 (local address 127.30.218.61:33527)
0806 10:59:27.643977 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:27.643984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:27.644013 (+ 29us) server_negotiation.cc:407] Connection header received
0806 10:59:27.644063 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:27.644070 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:27.644290 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:27.644453 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:27.645245 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:27.646507 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:27.647624 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:27.648187 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:27.648307 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:27.653802 (+ 5495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:27.653824 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:27.653848 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:27.653898 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:27.657489 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:27.658249 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:27.658258 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:27.658261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:27.658422 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:27.658768 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:27.658772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:27.658775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:27.659260 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:27.659475 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:27.659797 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 10:59:27.660055 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":450,"threads_started":1}
I20260806 10:59:27.663338 22661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:59:27.664520 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.664043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35999 (local address 127.0.0.1:43348)
0806 10:59:27.664227 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.664351 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 10:59:27.664804 22609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
I20260806 10:59:27.666505 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.666142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35999 (local address 127.0.0.1:43354)
0806 10:59:27.666289 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.666382 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 10:59:27.666741 22661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:35999: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.218.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.60.keytab.
Entry for principal kudu/127.30.218.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.60.keytab.
I20260806 10:59:27.694128 31592 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
/tmp/dist-test-taskT4s7YX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.60:35999
--webserver_interface=127.30.218.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.60.keytab
--principal=kudu/127.30.218.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.218.20:38547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.218.62:40601,127.30.218.61:33527,127.30.218.60:35999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 10:59:27.716117 22661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } attempt: 1
I20260806 10:59:27.717320 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.716836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35999 (local address 127.0.0.1:43360)
0806 10:59:27.717035 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.717156 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 10:59:27.718488 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.718093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35999 (local address 127.0.0.1:43366)
0806 10:59:27.718279 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.718367 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260806 10:59:27.718720 22661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:35999: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
I20260806 10:59:27.756686 22585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } attempt: 5
Aug 06 10:59:27 dist-test-slave-49h3 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013966, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for kudu/127.30.218.61@KRBTEST.COM
I20260806 10:59:27.770920 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.757640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33527 (local address 127.0.0.1:36502)
0806 10:59:27.757870 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.757893 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 10:59:27.758046 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:59:27.758420 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:59:27.758426 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:59:27.758874 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:59:27.759070 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:27.759080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:27.760098 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:27.760104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:27.760980 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:27.760995 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:27.761929 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:27.761936 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:27.762159 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:27.762824 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:59:27.762844 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:59:27.765029 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:59:27.768232 (+ 3203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:27.768238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:27.768243 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:27.768675 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:27.769083 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:27.769088 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:27.769093 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:27.769249 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:27.769874 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:59:27.769880 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:59:27.770049 (+ 169us) client_negotiation.cc:769] Sending connection context
0806 10:59:27.770182 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 10:59:27.770568 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 10:59:27.770911 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.757689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.30.218.61:33527)
0806 10:59:27.757883 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:27.757889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:27.757975 (+ 86us) server_negotiation.cc:407] Connection header received
0806 10:59:27.758179 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:27.758182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:27.758271 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:27.758360 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:27.759239 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:27.759948 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:27.761153 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:27.761749 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:27.761892 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:27.765286 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:27.765291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:27.765295 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:27.765334 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:27.768042 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:27.768814 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:27.768818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:27.768820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:27.768954 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:27.769368 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:27.769372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:27.769374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:27.769743 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:27.769894 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:27.770313 (+ 419us) server_negotiation.cc:299] Negotiation successful
0806 10:59:27.770560 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 10:59:27.772120 22585 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:59:27.773227 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.772792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35999 (local address 127.0.0.1:43382)
0806 10:59:27.772967 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.773055 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 10:59:27.774262 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.773894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35999 (local address 127.0.0.1:43384)
0806 10:59:27.774064 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.774136 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 10:59:27.774459 22585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:35999: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
I20260806 10:59:27.786948 22661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } attempt: 2
I20260806 10:59:27.788244 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.787718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35999 (local address 127.0.0.1:43394)
0806 10:59:27.787951 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.788053 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260806 10:59:27.789340 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.788962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35999 (local address 127.0.0.1:43396)
0806 10:59:27.789128 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.789203 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 10:59:27.789572 22661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:35999: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
I20260806 10:59:27.825724 22585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } attempt: 1
I20260806 10:59:27.826908 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.826435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35999 (local address 127.0.0.1:43400)
0806 10:59:27.826637 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.826742 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 10:59:27.827955 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.827607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35999 (local address 127.0.0.1:43416)
0806 10:59:27.827757 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.827828 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 10:59:27.828176 22585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:35999: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
I20260806 10:59:27.880836 22661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } attempt: 3
I20260806 10:59:27.882122 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.881629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35999 (local address 127.0.0.1:43428)
0806 10:59:27.881841 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.881936 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 10:59:27.883311 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.882841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35999 (local address 127.0.0.1:43430)
0806 10:59:27.883087 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.883175 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260806 10:59:27.883538 22661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:35999: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
I20260806 10:59:27.896404 22585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } attempt: 2
I20260806 10:59:27.897809 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.897261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35999 (local address 127.0.0.1:43442)
0806 10:59:27.897526 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.897628 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 10:59:27.898958 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:27.898607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35999 (local address 127.0.0.1:43456)
0806 10:59:27.898761 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:27.898843 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 10:59:27.899171 22585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:35999: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
I20260806 10:59:28.005424 22585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } attempt: 3
I20260806 10:59:28.006735 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:28.006200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35999 (local address 127.0.0.1:43470)
0806 10:59:28.006434 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:28.006535 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260806 10:59:28.007915 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:28.007511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35999 (local address 127.0.0.1:43478)
0806 10:59:28.007705 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:28.007786 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260806 10:59:28.008164 22585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:35999: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
I20260806 10:59:28.026789 22661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } attempt: 4
I20260806 10:59:28.028255 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:28.027609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35999 (local address 127.0.0.1:43494)
0806 10:59:28.027924 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:28.028055 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260806 10:59:28.029332 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:28.028974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35999 (local address 127.0.0.1:43508)
0806 10:59:28.029136 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:28.029215 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 10:59:28.029565 22661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.218.60:35999: Network error: Client connection negotiation failed: client connection to 127.30.218.60:35999: connect: Connection refused (error 111)
W20260806 10:59:28.058934 22670 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 10:59:28.059325 22670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 10:59:28.059424 22670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 10:59:28.059551 22670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 10:59:28.067245 22670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 10:59:28.067364 22670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 10:59:28.067466 22670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 10:59:28.067533 22670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 10:59:28.067616 22670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 10:59:28.078483 22670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.218.20:38547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.218.62:40601,127.30.218.61:33527,127.30.218.60:35999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/krb5kdc/kudu_127.30.218.60.keytab
--principal=kudu/127.30.218.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.218.60:35999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.218.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 09:53:15 UTC on ccycloud.quasar-vudqhy.root.comops.site
build id 4570067
ASAN enabled
I20260806 10:59:28.079938 22670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 10:59:28.081344 22670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 10:59:28.094576 22676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:28.094923 22678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 10:59:28.095484 22675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 10:59:28.096295 22670 server_base.cc:1034] running on GCE node
I20260806 10:59:28.097422 22670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 10:59:28.099160 22670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 10:59:28.100423 22670 hybrid_clock.cc:648] HybridClock initialized: now 1786013968100362 us; error 44 us; skew 500 ppm
I20260806 10:59:28.100776 22670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 10:59:28 dist-test-slave-49h3 krb5kdc[22503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013968, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 10:59:28.104113 22670 init.cc:377] Logged in from keytab as kudu/127.30.218.60@KRBTEST.COM (short username kudu)
I20260806 10:59:28.105729 22670 webserver.cc:460] Webserver started at http://127.30.218.60:39971/ using document root <none> and password file <none>
I20260806 10:59:28.106549 22670 fs_manager.cc:359] Metadata directory not provided
I20260806 10:59:28.106657 22670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 10:59:28.106989 22670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 10:59:28.110314 22670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-2/data/instance:
uuid: "f7dbb730a7964dd5b3adcfece616205e"
format_stamp: "Formatted at 2026-08-06 10:59:28 on dist-test-slave-49h3"
server_key: "7bd25f6bd4fcebfe9d966e02d36e5f06"
server_key_iv: "89f6b47b8949117fc3fa9732135b00e9"
server_key_version: "encryptionkey@0"
I20260806 10:59:28.111183 22670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-2/wal/instance:
uuid: "f7dbb730a7964dd5b3adcfece616205e"
format_stamp: "Formatted at 2026-08-06 10:59:28 on dist-test-slave-49h3"
server_key: "7bd25f6bd4fcebfe9d966e02d36e5f06"
server_key_iv: "89f6b47b8949117fc3fa9732135b00e9"
server_key_version: "encryptionkey@0"
I20260806 10:59:28.116990 22670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 10:59:28.121527 22685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 10:59:28.123917 22670 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260806 10:59:28.124127 22670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-2/data,/tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-2/wal
uuid: "f7dbb730a7964dd5b3adcfece616205e"
format_stamp: "Formatted at 2026-08-06 10:59:28 on dist-test-slave-49h3"
server_key: "7bd25f6bd4fcebfe9d966e02d36e5f06"
server_key_iv: "89f6b47b8949117fc3fa9732135b00e9"
server_key_version: "encryptionkey@0"
I20260806 10:59:28.124387 22670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 10:59:28.141657 22670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 10:59:28.145880 22670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 10:59:28.146179 22670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 10:59:28.164142 22670 rpc_server.cc:307] RPC server started. Bound to: 127.30.218.60:35999
I20260806 10:59:28.164175 22737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.218.60:35999 every 8 connection(s)
I20260806 10:59:28.165699 22670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-2/data/info.pb
I20260806 10:59:28.169866 22738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 10:59:28.170961 31592 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu as pid 22670
I20260806 10:59:28.171147 31592 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT4s7YX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786013589554914-31592-0/minicluster-data/master-2/wal/instance
I20260806 10:59:28.171605 31592 external_mini_cluster.cc:1428] Setting key 51f87541fed6c1d4b7bc4428f944752c
I20260806 10:59:28.180290 22738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40601 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:59:28.185503 22585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } attempt: 4
Aug 06 10:59:28 dist-test-slave-49h3 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.60@KRBTEST.COM
Aug 06 10:59:28 dist-test-slave-49h3 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013968, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.60@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
Aug 06 10:59:28 dist-test-slave-49h3 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013966, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.62@KRBTEST.COM for kudu/127.30.218.60@KRBTEST.COM
I20260806 10:59:28.200182 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:28.173765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43524 (local address 127.30.218.60:35999)
0806 10:59:28.174606 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:28.174622 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:28.174683 (+ 61us) server_negotiation.cc:407] Connection header received
0806 10:59:28.176769 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:28.176806 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:28.177490 (+ 684us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:28.178043 (+ 553us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:28.179633 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.181745 (+ 2112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.183257 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.184347 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.184563 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:28.190033 (+ 5470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:28.190049 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:28.190068 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:28.190128 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:28.194419 (+ 4291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:28.195369 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:28.195377 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:28.195385 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:28.195561 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:28.196013 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:28.196017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:28.196020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:28.196980 (+ 960us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:28.197317 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:28.197918 (+ 601us) server_negotiation.cc:299] Negotiation successful
0806 10:59:28.198458 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"spinlock_wait_cycles":48384,"thread_start_us":325,"threads_started":1}
I20260806 10:59:28.201959 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:28.183674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.62:40601 (local address 127.0.0.1:36068)
0806 10:59:28.184757 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:28.184810 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 10:59:28.184967 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:59:28.185465 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:59:28.185474 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:59:28.186444 (+ 970us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:59:28.186863 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:28.186880 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.188273 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.188280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:28.190135 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:28.190149 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.191201 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.191207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:28.191403 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:28.192154 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:59:28.192182 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:59:28.194738 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:59:28.198663 (+ 3925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:28.198677 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:28.198685 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:28.199187 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:28.199584 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:28.199597 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:28.199603 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:28.199745 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:28.200688 (+ 943us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:59:28.200704 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:59:28.200880 (+ 176us) client_negotiation.cc:769] Sending connection context
0806 10:59:28.201075 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 10:59:28.201425 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":31,"thread_start_us":645,"threads_started":1}
I20260806 10:59:28.203068 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:28.183761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36068 (local address 127.30.218.62:40601)
0806 10:59:28.184336 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:28.184342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:28.184905 (+ 563us) server_negotiation.cc:407] Connection header received
0806 10:59:28.185085 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:28.185090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:28.185186 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:28.185296 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:28.187029 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.188095 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.190315 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.190773 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.190908 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:28.194976 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:28.194983 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:28.194989 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:28.195052 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:28.198521 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:28.199329 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:28.199335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:28.199339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:28.199474 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:28.199867 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:28.199872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:28.199876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:28.200450 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:28.200616 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:28.201243 (+ 627us) server_negotiation.cc:299] Negotiation successful
0806 10:59:28.201507 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":423,"thread_start_us":290,"threads_started":1}
I20260806 10:59:28.203115 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:28.186185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35999 (local address 127.0.0.1:43540)
0806 10:59:28.186455 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:28.186475 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 10:59:28.186577 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:59:28.187449 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:59:28.187454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:59:28.187982 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:59:28.188197 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:28.188208 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.189715 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.189721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:28.191149 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:28.191169 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.192163 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.192170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:28.192400 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:28.192981 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:59:28.193004 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:59:28.195765 (+ 2761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:59:28.200244 (+ 4479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:28.200249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:28.200253 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:28.200798 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:28.201281 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:28.201285 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:28.201289 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:28.201439 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:28.202155 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:59:28.202160 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:59:28.202360 (+ 200us) client_negotiation.cc:769] Sending connection context
0806 10:59:28.202484 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 10:59:28.202707 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260806 10:59:28.204152 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:28.186327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43540 (local address 127.30.218.60:35999)
0806 10:59:28.187124 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:28.187130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:28.187152 (+ 22us) server_negotiation.cc:407] Connection header received
0806 10:59:28.187202 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:28.187207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:28.187281 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:28.187384 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:28.188342 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.189558 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.191332 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.191936 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.192114 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:28.195996 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:28.196004 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:28.196008 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:28.196048 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:28.200047 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:28.200974 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:28.200979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:28.200982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:28.201134 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:28.201619 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:28.201624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:28.201627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:28.202011 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:28.203388 (+ 1377us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:28.203521 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 10:59:28.203762 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":661,"thread_start_us":482,"threads_started":1}
I20260806 10:59:28.204551 22738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:59:28.211177 22585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c: Bootstrap starting.
Aug 06 10:59:28 dist-test-slave-49h3 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013968, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.60@KRBTEST.COM for kudu/127.30.218.61@KRBTEST.COM
Aug 06 10:59:28 dist-test-slave-49h3 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.62@KRBTEST.COM
I20260806 10:59:28.215370 22585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c: Neither blocks nor log segments found. Creating new log.
I20260806 10:59:28.216995 22585 log.cc:824] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c: Log is configured to *not* fsync() on all Append() calls
I20260806 10:59:28.220484 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:28.205283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.61:33527 (local address 127.0.0.1:36506)
0806 10:59:28.205481 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:28.205500 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 10:59:28.205596 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:59:28.206472 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:59:28.206477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:59:28.206975 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:59:28.207164 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:28.207177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.208323 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.208329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:28.209521 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:28.209539 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.210772 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.210780 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:28.211008 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:28.211640 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:59:28.211666 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:59:28.214476 (+ 2810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:59:28.217839 (+ 3363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:28.217846 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:28.217850 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:28.218284 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:28.218715 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:28.218720 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:28.218725 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:28.218888 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:28.219459 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:59:28.219464 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:59:28.219633 (+ 169us) client_negotiation.cc:769] Sending connection context
0806 10:59:28.219772 (+ 139us) client_negotiation.cc:240] Negotiation successful
0806 10:59:28.220112 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 10:59:28.220515 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:28.205771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36506 (local address 127.30.218.61:33527)
0806 10:59:28.205974 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:28.205980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:28.205999 (+ 19us) server_negotiation.cc:407] Connection header received
0806 10:59:28.206074 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:28.206078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:28.206160 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:28.206270 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:28.207351 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.208160 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.209690 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.210314 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.210456 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:28.214730 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:28.214736 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:28.214739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:28.214780 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:28.217668 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:28.218432 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:28.218437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:28.218439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:28.218591 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:28.219015 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:28.219019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:28.219021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:28.219321 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:28.219482 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:28.219911 (+ 429us) server_negotiation.cc:299] Negotiation successful
0806 10:59:28.220135 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 10:59:28.222012 22738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } has no permanent_uuid. Determining permanent_uuid...
I20260806 10:59:28.223593 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:28.204277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36078 (local address 127.30.218.62:40601)
0806 10:59:28.204540 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:28.204546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:28.204563 (+ 17us) server_negotiation.cc:407] Connection header received
0806 10:59:28.204610 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:28.204614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:28.204685 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:28.204782 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:28.206334 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.207258 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.209052 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.209755 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.209867 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:28.215674 (+ 5807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:28.215681 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:28.215685 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:28.215728 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:28.219420 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:28.220569 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:28.220576 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:28.220580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:28.220768 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:28.221291 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:28.221297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:28.221300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:28.221773 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:28.222099 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:28.222999 (+ 900us) server_negotiation.cc:299] Negotiation successful
0806 10:59:28.223265 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260806 10:59:28.224720 22585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c: No bootstrap required, opened a new log
Aug 06 10:59:28 dist-test-slave-49h3 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013968, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.60@KRBTEST.COM for kudu/127.30.218.60@KRBTEST.COM
Aug 06 10:59:28 dist-test-slave-49h3 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.218.61@KRBTEST.COM
I20260806 10:59:28.236295 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:28.222959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35999 (local address 127.0.0.1:43550)
0806 10:59:28.223634 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:28.223651 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 10:59:28.223798 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:59:28.224226 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:59:28.224229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:59:28.224650 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:59:28.224818 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:28.224830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.226071 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.226077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:28.226948 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:28.226960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.227691 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.227696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:28.227863 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:28.228500 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:59:28.228525 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:59:28.230776 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:59:28.233914 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:28.233921 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:28.233926 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:28.234288 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:28.234641 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:28.234647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:28.234651 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:28.234790 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:28.235284 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:59:28.235288 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:59:28.235446 (+ 158us) client_negotiation.cc:769] Sending connection context
0806 10:59:28.235565 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 10:59:28.235900 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591}
I20260806 10:59:28.236289 22585 raft_consensus.cc:348] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea499cc79b3491ab5a5d13802e1a67c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40601 } } peers { permanent_uuid: "30b5b7cf19a94117ac993d27f6862fc5" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } } peers { permanent_uuid: "f7dbb730a7964dd5b3adcfece616205e" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } }
I20260806 10:59:28.236702 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:28.223356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43550 (local address 127.30.218.60:35999)
0806 10:59:28.223532 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:28.223537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:28.223707 (+ 170us) server_negotiation.cc:407] Connection header received
0806 10:59:28.223912 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:28.223916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:28.223985 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:28.224049 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:28.224937 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.225944 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.227088 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.227534 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.227642 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:28.230967 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:28.230972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:28.230976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:28.231008 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:28.233794 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:28.234399 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:28.234404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:28.234407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:28.234527 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:28.234887 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:28.234892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:28.234895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:28.235166 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:28.235318 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:28.235686 (+ 368us) server_negotiation.cc:299] Negotiation successful
0806 10:59:28.236057 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":149,"server-negotiator.queue_time_us":57}
I20260806 10:59:28.236779 22585 raft_consensus.cc:729] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dea499cc79b3491ab5a5d13802e1a67c, State: Initialized, Role: FOLLOWER
I20260806 10:59:28.237402 22585 consensus_queue.cc:260] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea499cc79b3491ab5a5d13802e1a67c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40601 } } peers { permanent_uuid: "30b5b7cf19a94117ac993d27f6862fc5" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } } peers { permanent_uuid: "f7dbb730a7964dd5b3adcfece616205e" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } }
I20260806 10:59:28.238843 22747 sys_catalog.cc:455] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea499cc79b3491ab5a5d13802e1a67c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40601 } } peers { permanent_uuid: "30b5b7cf19a94117ac993d27f6862fc5" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } } peers { permanent_uuid: "f7dbb730a7964dd5b3adcfece616205e" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } } }
I20260806 10:59:28.239662 22747 sys_catalog.cc:458] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:59:28.240032 22585 sys_catalog.cc:564] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:59:28.243201 22738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7dbb730a7964dd5b3adcfece616205e: Bootstrap starting.
I20260806 10:59:28.244155 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:28.225592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36518 (local address 127.30.218.61:33527)
0806 10:59:28.225783 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:28.225788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:28.225809 (+ 21us) server_negotiation.cc:407] Connection header received
0806 10:59:28.225922 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:28.225927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:28.226006 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:28.227819 (+ 1813us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:28.228058 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.229402 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.231352 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.232003 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.232778 (+ 775us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:28.237655 (+ 4877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:28.237663 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:28.237668 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:28.237715 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:28.240891 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:28.241926 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:28.241933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:28.241936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:28.242106 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:28.242632 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:28.242636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:28.242638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:28.242945 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:28.243232 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:28.243601 (+ 369us) server_negotiation.cc:299] Negotiation successful
0806 10:59:28.243848 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 10:59:28.248759 22738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7dbb730a7964dd5b3adcfece616205e: Neither blocks nor log segments found. Creating new log.
I20260806 10:59:28.249584 31592 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 10:59:28.250737 22738 log.cc:824] T 00000000000000000000000000000000 P f7dbb730a7964dd5b3adcfece616205e: Log is configured to *not* fsync() on all Append() calls
I20260806 10:59:28.252792 22559 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "dea499cc79b3491ab5a5d13802e1a67c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36078
I20260806 10:59:28.253239 22559 raft_consensus.cc:482] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 10:59:28.253495 22559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:59:28.259137 22738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7dbb730a7964dd5b3adcfece616205e: No bootstrap required, opened a new log
I20260806 10:59:28.259002 22559 raft_consensus.cc:504] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea499cc79b3491ab5a5d13802e1a67c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40601 } } peers { permanent_uuid: "30b5b7cf19a94117ac993d27f6862fc5" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } } peers { permanent_uuid: "f7dbb730a7964dd5b3adcfece616205e" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } }
I20260806 10:59:28.261035 22559 leader_election.cc:288] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c [CANDIDATE]: Term 1 election: Requested vote from peers 30b5b7cf19a94117ac993d27f6862fc5 (127.30.218.61:33527), f7dbb730a7964dd5b3adcfece616205e (127.30.218.60:35999)
I20260806 10:59:28.254422 22516 master_runner.cc:186] Error getting master registration for 127.30.218.61:33527: OK, instance_id { permanent_uuid: "30b5b7cf19a94117ac993d27f6862fc5" instance_seqno: 1786013967599733 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 10:59:28.268535 22516 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 10:59:28.271987 22758 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 10:59:28.272186 22758 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 10:59:28.276392 22738 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7dbb730a7964dd5b3adcfece616205e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea499cc79b3491ab5a5d13802e1a67c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40601 } } peers { permanent_uuid: "30b5b7cf19a94117ac993d27f6862fc5" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } } peers { permanent_uuid: "f7dbb730a7964dd5b3adcfece616205e" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } }
I20260806 10:59:28.276933 22738 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7dbb730a7964dd5b3adcfece616205e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7dbb730a7964dd5b3adcfece616205e, State: Initialized, Role: FOLLOWER
I20260806 10:59:28.277948 22738 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7dbb730a7964dd5b3adcfece616205e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea499cc79b3491ab5a5d13802e1a67c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40601 } } peers { permanent_uuid: "30b5b7cf19a94117ac993d27f6862fc5" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } } peers { permanent_uuid: "f7dbb730a7964dd5b3adcfece616205e" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } }
W20260806 10:59:28.278481 22537 tablet.cc:2369] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:59:28.287518 22738 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7dbb730a7964dd5b3adcfece616205e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:59:28.287719 22713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dea499cc79b3491ab5a5d13802e1a67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f7dbb730a7964dd5b3adcfece616205e"
I20260806 10:59:28.288487 22713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7dbb730a7964dd5b3adcfece616205e [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:59:28.294308 22713 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7dbb730a7964dd5b3adcfece616205e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dea499cc79b3491ab5a5d13802e1a67c in term 1.
I20260806 10:59:28.295174 22532 leader_election.cc:302] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dea499cc79b3491ab5a5d13802e1a67c, f7dbb730a7964dd5b3adcfece616205e; no voters:
I20260806 10:59:28.295845 22747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 10:59:28.285089 22764 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7dbb730a7964dd5b3adcfece616205e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea499cc79b3491ab5a5d13802e1a67c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40601 } } peers { permanent_uuid: "30b5b7cf19a94117ac993d27f6862fc5" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } } peers { permanent_uuid: "f7dbb730a7964dd5b3adcfece616205e" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } } }
I20260806 10:59:28.301102 22764 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7dbb730a7964dd5b3adcfece616205e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:59:28.315281 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:28.292156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43564 (local address 127.30.218.60:35999)
0806 10:59:28.292396 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:28.292402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:28.295491 (+ 3089us) server_negotiation.cc:407] Connection header received
0806 10:59:28.295765 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:28.295770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:28.295876 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:28.295955 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:28.298391 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.299499 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.301374 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.302173 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.302284 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:28.306554 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:28.306563 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:28.306569 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:28.306624 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:28.310291 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:28.311839 (+ 1548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:28.311847 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:28.311852 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:28.312079 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:28.312887 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:28.312893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:28.312896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:28.313302 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:28.313614 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:28.314608 (+ 994us) server_negotiation.cc:299] Negotiation successful
0806 10:59:28.314887 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260806 10:59:28.323988 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:28.291573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36080 (local address 127.30.218.62:40601)
0806 10:59:28.291798 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:28.291806 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:28.292452 (+ 646us) server_negotiation.cc:407] Connection header received
0806 10:59:28.292726 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:28.292733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:28.292820 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:28.292922 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:28.296524 (+ 3602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.297811 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.299953 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.300696 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.300831 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:28.312973 (+ 12142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:28.312981 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:28.312985 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:28.313038 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:28.318123 (+ 5085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:28.319082 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:28.319089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:28.319093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:28.319419 (+ 326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:28.320141 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:28.320146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:28.320150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:28.320602 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:28.320913 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:28.321748 (+ 835us) server_negotiation.cc:299] Negotiation successful
0806 10:59:28.322297 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 10:59:28.325264 22747 raft_consensus.cc:686] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c [term 1 LEADER]: Becoming Leader. State: Replica: dea499cc79b3491ab5a5d13802e1a67c, State: Running, Role: LEADER
I20260806 10:59:28.326107 22747 consensus_queue.cc:237] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea499cc79b3491ab5a5d13802e1a67c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40601 } } peers { permanent_uuid: "30b5b7cf19a94117ac993d27f6862fc5" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } } peers { permanent_uuid: "f7dbb730a7964dd5b3adcfece616205e" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } }
I20260806 10:59:28.328759 22661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } attempt: 5
I20260806 10:59:28.333407 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:28.292189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36524 (local address 127.30.218.61:33527)
0806 10:59:28.292555 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:28.292561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:28.308104 (+ 15543us) server_negotiation.cc:407] Connection header received
0806 10:59:28.308981 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:28.308989 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:28.309088 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:28.309169 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:28.311393 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.312579 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.316259 (+ 3680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.316980 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.317097 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:28.323681 (+ 6584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:28.323690 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:28.323696 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:28.323746 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:28.327684 (+ 3938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:28.329674 (+ 1990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:28.329681 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:28.329684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:28.329899 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:28.330888 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:28.330894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:28.330898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:28.331324 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:28.331601 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:28.332729 (+ 1128us) server_negotiation.cc:299] Negotiation successful
0806 10:59:28.333016 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238}
Aug 06 10:59:28 dist-test-slave-49h3 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786013967, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.218.61@KRBTEST.COM for kudu/127.30.218.60@KRBTEST.COM
I20260806 10:59:28.342207 22770 sys_catalog.cc:455] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c [sys.catalog]: SysCatalogTable state changed. Reason: New leader dea499cc79b3491ab5a5d13802e1a67c. Latest consensus state: current_term: 1 leader_uuid: "dea499cc79b3491ab5a5d13802e1a67c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea499cc79b3491ab5a5d13802e1a67c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40601 } } peers { permanent_uuid: "30b5b7cf19a94117ac993d27f6862fc5" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } } peers { permanent_uuid: "f7dbb730a7964dd5b3adcfece616205e" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } } }
I20260806 10:59:28.342667 22770 sys_catalog.cc:458] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:28.343681 22776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 10:59:28.347172 22776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 10:59:28.348026 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:28.329653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43576 (local address 127.30.218.60:35999)
0806 10:59:28.329888 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:28.329893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:28.329911 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:59:28.330034 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:28.330040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:28.330122 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:28.330223 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:28.331283 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.332412 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.334382 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.335355 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.335504 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:28.340827 (+ 5323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 10:59:28.340836 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 10:59:28.340842 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 10:59:28.340890 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 10:59:28.344556 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:28.345574 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:28.345581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:28.345585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:28.345752 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 10:59:28.346280 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 10:59:28.346285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 10:59:28.346288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 10:59:28.346700 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 10:59:28.346842 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:28.347413 (+ 571us) server_negotiation.cc:299] Negotiation successful
0806 10:59:28.347678 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260806 10:59:28.348958 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:28.329425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.218.60:35999 (local address 127.0.0.1:43576)
0806 10:59:28.329742 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 10:59:28.329763 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 10:59:28.329885 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 10:59:28.330253 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 10:59:28.330257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 10:59:28.330855 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0806 10:59:28.331113 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:28.331127 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.332567 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.332573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:28.334214 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 10:59:28.334235 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 10:59:28.335530 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 10:59:28.335538 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 10:59:28.335842 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:28.336747 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 10:59:28.336774 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 10:59:28.340541 (+ 3767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 10:59:28.344787 (+ 4246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:28.344792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:28.344796 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:28.345366 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:28.345918 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 10:59:28.345923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 10:59:28.345927 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 10:59:28.346132 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 10:59:28.346937 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 10:59:28.346942 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 10:59:28.347119 (+ 177us) client_negotiation.cc:769] Sending connection context
0806 10:59:28.348222 (+ 1103us) client_negotiation.cc:240] Negotiation successful
0806 10:59:28.348585 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260806 10:59:28.358991 22661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30b5b7cf19a94117ac993d27f6862fc5: Bootstrap starting.
I20260806 10:59:28.366649 22713 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7dbb730a7964dd5b3adcfece616205e [term 1 FOLLOWER]: Refusing update from remote peer dea499cc79b3491ab5a5d13802e1a67c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:59:28.367676 22747 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7dbb730a7964dd5b3adcfece616205e" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:59:28.382465 22661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30b5b7cf19a94117ac993d27f6862fc5: Neither blocks nor log segments found. Creating new log.
I20260806 10:59:28.382642 22764 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7dbb730a7964dd5b3adcfece616205e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dea499cc79b3491ab5a5d13802e1a67c. Latest consensus state: current_term: 1 leader_uuid: "dea499cc79b3491ab5a5d13802e1a67c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea499cc79b3491ab5a5d13802e1a67c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40601 } } peers { permanent_uuid: "30b5b7cf19a94117ac993d27f6862fc5" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } } peers { permanent_uuid: "f7dbb730a7964dd5b3adcfece616205e" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } } }
I20260806 10:59:28.382973 22764 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7dbb730a7964dd5b3adcfece616205e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:59:28.384220 22661 log.cc:824] T 00000000000000000000000000000000 P 30b5b7cf19a94117ac993d27f6862fc5: Log is configured to *not* fsync() on all Append() calls
I20260806 10:59:28.389294 22776 catalog_manager.cc:1347] Generated new cluster ID: 873f0ad059154b37920b7f1381e432fe
I20260806 10:59:28.389410 22776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 10:59:28.389962 22661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30b5b7cf19a94117ac993d27f6862fc5: No bootstrap required, opened a new log
W20260806 10:59:28.398339 22692 tablet.cc:2369] T 00000000000000000000000000000000 P f7dbb730a7964dd5b3adcfece616205e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 10:59:28.409727 22770 sys_catalog.cc:455] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dea499cc79b3491ab5a5d13802e1a67c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea499cc79b3491ab5a5d13802e1a67c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40601 } } peers { permanent_uuid: "30b5b7cf19a94117ac993d27f6862fc5" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } } peers { permanent_uuid: "f7dbb730a7964dd5b3adcfece616205e" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } } }
I20260806 10:59:28.410195 22770 sys_catalog.cc:458] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:28.410099 22661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30b5b7cf19a94117ac993d27f6862fc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea499cc79b3491ab5a5d13802e1a67c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40601 } } peers { permanent_uuid: "30b5b7cf19a94117ac993d27f6862fc5" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } } peers { permanent_uuid: "f7dbb730a7964dd5b3adcfece616205e" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } }
I20260806 10:59:28.410908 22661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30b5b7cf19a94117ac993d27f6862fc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30b5b7cf19a94117ac993d27f6862fc5, State: Initialized, Role: FOLLOWER
I20260806 10:59:28.411972 22661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30b5b7cf19a94117ac993d27f6862fc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea499cc79b3491ab5a5d13802e1a67c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40601 } } peers { permanent_uuid: "30b5b7cf19a94117ac993d27f6862fc5" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } } peers { permanent_uuid: "f7dbb730a7964dd5b3adcfece616205e" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } }
I20260806 10:59:28.414652 22764 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7dbb730a7964dd5b3adcfece616205e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dea499cc79b3491ab5a5d13802e1a67c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea499cc79b3491ab5a5d13802e1a67c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40601 } } peers { permanent_uuid: "30b5b7cf19a94117ac993d27f6862fc5" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } } peers { permanent_uuid: "f7dbb730a7964dd5b3adcfece616205e" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } } }
I20260806 10:59:28.414942 22764 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7dbb730a7964dd5b3adcfece616205e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:59:28.387212 22670 master_runner.cc:186] Error getting master registration for 127.30.218.61:33527: OK, instance_id { permanent_uuid: "30b5b7cf19a94117ac993d27f6862fc5" instance_seqno: 1786013967599733 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 10:59:28.417728 22670 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 10:59:28.418455 22791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30b5b7cf19a94117ac993d27f6862fc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea499cc79b3491ab5a5d13802e1a67c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40601 } } peers { permanent_uuid: "30b5b7cf19a94117ac993d27f6862fc5" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } } peers { permanent_uuid: "f7dbb730a7964dd5b3adcfece616205e" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } } }
I20260806 10:59:28.419057 22791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30b5b7cf19a94117ac993d27f6862fc5 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:59:28.420452 22661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30b5b7cf19a94117ac993d27f6862fc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 10:59:28.422520 22636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dea499cc79b3491ab5a5d13802e1a67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "30b5b7cf19a94117ac993d27f6862fc5"
I20260806 10:59:28.423120 22636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30b5b7cf19a94117ac993d27f6862fc5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 10:59:28.426862 22636 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 30b5b7cf19a94117ac993d27f6862fc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dea499cc79b3491ab5a5d13802e1a67c in term 1.
I20260806 10:59:28.428649 22784 catalog_manager.cc:1259] Loaded cluster ID: 873f0ad059154b37920b7f1381e432fe
I20260806 10:59:28.428767 22784 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f7dbb730a7964dd5b3adcfece616205e: loading cluster ID for follower catalog manager: success
I20260806 10:59:28.428745 22635 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 30b5b7cf19a94117ac993d27f6862fc5 [term 1 FOLLOWER]: Refusing update from remote peer dea499cc79b3491ab5a5d13802e1a67c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 10:59:28.432749 22790 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30b5b7cf19a94117ac993d27f6862fc5" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 10:59:28.434978 22776 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 10:59:28.436985 22776 catalog_manager.cc:1504] Loading token signing keys...
W20260806 10:59:28.441370 22784 catalog_manager.cc:1576] T 00000000000000000000000000000000 P f7dbb730a7964dd5b3adcfece616205e: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260806 10:59:28.441568 22784 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260806 10:59:28.465269 22776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c: Generated new TSK 0
I20260806 10:59:28.466717 22776 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 10:59:28.521535 22791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30b5b7cf19a94117ac993d27f6862fc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dea499cc79b3491ab5a5d13802e1a67c. Latest consensus state: current_term: 1 leader_uuid: "dea499cc79b3491ab5a5d13802e1a67c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea499cc79b3491ab5a5d13802e1a67c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40601 } } peers { permanent_uuid: "30b5b7cf19a94117ac993d27f6862fc5" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } } peers { permanent_uuid: "f7dbb730a7964dd5b3adcfece616205e" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } } }
I20260806 10:59:28.521905 22791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30b5b7cf19a94117ac993d27f6862fc5 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:59:28.521922 22747 sys_catalog.cc:455] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dea499cc79b3491ab5a5d13802e1a67c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea499cc79b3491ab5a5d13802e1a67c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40601 } } peers { permanent_uuid: "30b5b7cf19a94117ac993d27f6862fc5" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } } peers { permanent_uuid: "f7dbb730a7964dd5b3adcfece616205e" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } } }
I20260806 10:59:28.522253 22747 sys_catalog.cc:458] T 00000000000000000000000000000000 P dea499cc79b3491ab5a5d13802e1a67c [sys.catalog]: This master's current role is: LEADER
I20260806 10:59:28.523366 22801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30b5b7cf19a94117ac993d27f6862fc5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dea499cc79b3491ab5a5d13802e1a67c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea499cc79b3491ab5a5d13802e1a67c" member_type: VOTER last_known_addr { host: "127.30.218.62" port: 40601 } } peers { permanent_uuid: "30b5b7cf19a94117ac993d27f6862fc5" member_type: VOTER last_known_addr { host: "127.30.218.61" port: 33527 } } peers { permanent_uuid: "f7dbb730a7964dd5b3adcfece616205e" member_type: VOTER last_known_addr { host: "127.30.218.60" port: 35999 } } }
I20260806 10:59:28.523823 22801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30b5b7cf19a94117ac993d27f6862fc5 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 10:59:28.550736 22810 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260806 10:59:28.588883 22742 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:28.585031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36090 (local address 127.30.218.62:40601)
0806 10:59:28.585276 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:28.585283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:28.585894 (+ 611us) server_negotiation.cc:407] Connection header received
0806 10:59:28.586255 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:28.586262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:28.586358 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:28.586443 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:28.588534 (+ 2091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36090: BlockingRecv error: recv got EOF from 127.0.0.1:36090 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260806 10:59:28.591746 22745 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:28.586944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43584 (local address 127.30.218.60:35999)
0806 10:59:28.587159 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:28.587166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:28.589835 (+ 2669us) server_negotiation.cc:407] Connection header received
0806 10:59:28.590036 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:28.590043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:28.590155 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:28.590273 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:28.591459 (+ 1186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43584: BlockingRecv error: recv got EOF from 127.0.0.1:43584 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 10:59:28.592828 22666 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:28.586729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36536 (local address 127.30.218.61:33527)
0806 10:59:28.587195 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:28.587200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:28.590525 (+ 3325us) server_negotiation.cc:407] Connection header received
0806 10:59:28.590819 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:28.590826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:28.590933 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:28.591009 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:28.592558 (+ 1549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36536: BlockingRecv error: recv got EOF from 127.0.0.1:36536 (error 108)
Metrics: {"server-negotiator.queue_time_us":294}
W20260806 10:59:28.604911 22666 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:28.602330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36546 (local address 127.30.218.61:33527)
0806 10:59:28.602536 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:28.602542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:28.603281 (+ 739us) server_negotiation.cc:407] Connection header received
0806 10:59:28.603529 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:28.603534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:28.603643 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:28.603741 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:28.604600 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36546: BlockingRecv error: recv got EOF from 127.0.0.1:36546 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260806 10:59:28.645124 22666 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:28.642824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36562 (local address 127.30.218.61:33527)
0806 10:59:28.643054 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:28.643061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:28.643647 (+ 586us) server_negotiation.cc:407] Connection header received
0806 10:59:28.643851 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:28.643856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:28.643938 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:28.644029 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:28.644894 (+ 865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36562: BlockingRecv error: recv got EOF from 127.0.0.1:36562 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260806 10:59:28.652796 22666 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:28.650520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36566 (local address 127.30.218.61:33527)
0806 10:59:28.650735 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:28.650743 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:28.651280 (+ 537us) server_negotiation.cc:407] Connection header received
0806 10:59:28.651512 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:28.651518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:28.651614 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:28.651706 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:28.652513 (+ 807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36566: BlockingRecv error: recv got EOF from 127.0.0.1:36566 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260806 10:59:28.662740 22666 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:28.660234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36582 (local address 127.30.218.61:33527)
0806 10:59:28.660521 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:28.660528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:28.661122 (+ 594us) server_negotiation.cc:407] Connection header received
0806 10:59:28.661346 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:28.661352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:28.661468 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:28.661595 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:28.662475 (+ 880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36582: BlockingRecv error: recv got EOF from 127.0.0.1:36582 (error 108)
Metrics: {"server-negotiator.queue_time_us":141}
W20260806 10:59:28.676604 22666 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:28.674774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36592 (local address 127.30.218.61:33527)
0806 10:59:28.674964 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:28.674970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:28.675045 (+ 75us) server_negotiation.cc:407] Connection header received
0806 10:59:28.675355 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:28.675359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:28.675438 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:28.675530 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:28.676356 (+ 826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36592: BlockingRecv error: recv got EOF from 127.0.0.1:36592 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260806 10:59:28.698689 22666 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:28.696330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36602 (local address 127.30.218.61:33527)
0806 10:59:28.696559 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:28.696567 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:28.697152 (+ 585us) server_negotiation.cc:407] Connection header received
0806 10:59:28.697362 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:28.697366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:28.697475 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:28.697592 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:28.698434 (+ 842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36602: BlockingRecv error: recv got EOF from 127.0.0.1:36602 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260806 10:59:28.736850 22666 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:28.734512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.30.218.61:33527)
0806 10:59:28.734755 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:28.734762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:28.735341 (+ 579us) server_negotiation.cc:407] Connection header received
0806 10:59:28.735556 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:28.735561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:28.735654 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:28.735749 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:28.736565 (+ 816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36618: BlockingRecv error: recv got EOF from 127.0.0.1:36618 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260806 10:59:28.810148 22666 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:28.807498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36626 (local address 127.30.218.61:33527)
0806 10:59:28.807706 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:28.807713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:28.808443 (+ 730us) server_negotiation.cc:407] Connection header received
0806 10:59:28.808671 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:28.808677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:28.808781 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:28.808902 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:28.809914 (+ 1012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36626: BlockingRecv error: recv got EOF from 127.0.0.1:36626 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260806 10:59:28.944962 22666 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:28.942926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36642 (local address 127.30.218.61:33527)
0806 10:59:28.943155 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:28.943161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:28.943483 (+ 322us) server_negotiation.cc:407] Connection header received
0806 10:59:28.943700 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:28.943704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:28.943783 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:28.943876 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:28.944677 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36642: BlockingRecv error: recv got EOF from 127.0.0.1:36642 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260806 10:59:29.209183 22666 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 10:59:29.207402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36648 (local address 127.30.218.61:33527)
0806 10:59:29.207608 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:29.207613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:29.207876 (+ 263us) server_negotiation.cc:407] Connection header received
0806 10:59:29.208073 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:29.208078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:29.208157 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:29.208257 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 10:59:29.208961 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36648: BlockingRecv error: recv got EOF from 127.0.0.1:36648 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 10:59:29.445739 22784 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f7dbb730a7964dd5b3adcfece616205e: acquiring CA information for follower catalog manager: success
I20260806 10:59:29.448057 22784 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f7dbb730a7964dd5b3adcfece616205e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 10:59:29.555244 22810 catalog_manager.cc:1259] Loaded cluster ID: 873f0ad059154b37920b7f1381e432fe
I20260806 10:59:29.555384 22810 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 30b5b7cf19a94117ac993d27f6862fc5: loading cluster ID for follower catalog manager: success
I20260806 10:59:29.558213 22810 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 30b5b7cf19a94117ac993d27f6862fc5: acquiring CA information for follower catalog manager: success
I20260806 10:59:29.560245 22810 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 30b5b7cf19a94117ac993d27f6862fc5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 10:59:29.733755 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:29.725622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36664 (local address 127.30.218.61:33527)
0806 10:59:29.726958 (+ 1336us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:29.726970 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:29.727525 (+ 555us) server_negotiation.cc:407] Connection header received
0806 10:59:29.727695 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:29.727701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:29.727920 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:29.728109 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:59:29.728721 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:29.729706 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:29.731166 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:29.731668 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:29.731793 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:29.732387 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:59:29.733008 (+ 621us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:59:29.733174 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:29.733296 (+ 122us) server_negotiation.cc:299] Negotiation successful
0806 10:59:29.733398 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1162,"thread_start_us":953,"threads_started":1}
I20260806 10:59:29.749941 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:29.742748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43596 (local address 127.30.218.60:35999)
0806 10:59:29.743536 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:29.743540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:29.743558 (+ 18us) server_negotiation.cc:407] Connection header received
0806 10:59:29.743660 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:29.743664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:29.743824 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:29.743975 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:59:29.744452 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:29.745231 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:29.747412 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:29.747834 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:29.747943 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:29.748600 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:59:29.749174 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:59:29.749302 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:29.749534 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 10:59:29.749621 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":572,"threads_started":1}
I20260806 10:59:29.777251 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:29.757830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43606 (local address 127.30.218.60:35999)
0806 10:59:29.758019 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:29.758025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:29.769560 (+ 11535us) server_negotiation.cc:407] Connection header received
0806 10:59:29.769936 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:29.769943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:29.770045 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:29.770114 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:59:29.771159 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:29.772348 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:29.774797 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:29.775475 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:29.775584 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:29.776347 (+ 763us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:59:29.776495 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:59:29.776559 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:29.776812 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 10:59:29.776912 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 10:59:29.778043 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:29.757543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36100 (local address 127.30.218.62:40601)
0806 10:59:29.768593 (+ 11050us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:29.768600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:29.769308 (+ 708us) server_negotiation.cc:407] Connection header received
0806 10:59:29.769541 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:29.769548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:29.769848 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:29.770050 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:59:29.771186 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:29.772440 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:29.775019 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:29.775613 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:29.775748 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:29.776509 (+ 761us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:59:29.777025 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:59:29.777134 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:29.777624 (+ 490us) server_negotiation.cc:299] Negotiation successful
0806 10:59:29.777739 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10888,"thread_start_us":10771,"threads_started":1}
I20260806 10:59:29.783330 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 10:59:29.757335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36668 (local address 127.30.218.61:33527)
0806 10:59:29.757571 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 10:59:29.757577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 10:59:29.775237 (+ 17660us) server_negotiation.cc:407] Connection header received
0806 10:59:29.775803 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 10:59:29.775809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 10:59:29.775920 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 10:59:29.775989 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 10:59:29.777194 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:29.778399 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:29.780724 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 10:59:29.781431 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 10:59:29.781623 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 10:59:29.782307 (+ 684us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 10:59:29.782469 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 10:59:29.782578 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 10:59:29.782750 (+ 172us) server_negotiation.cc:299] Negotiation successful
0806 10:59:29.782850 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 10:59:29.783602 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 22516
I20260806 10:59:29.784741 22580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:59:29.929888 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 22516
I20260806 10:59:29.955642 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 22592
I20260806 10:59:29.956797 22656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:59:30.065259 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 22592
I20260806 10:59:30.085750 31592 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu pid 22670
I20260806 10:59:30.087019 22733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 10:59:30.200882 31592 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT4s7YX/build/asan/bin/kudu with pid 22670
2026-08-06T10:59:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5067 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10149 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (380604 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST